{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 个性化推荐\n",
    "本项目使用文本卷积神经网络，并使用[`MovieLens`](https://grouplens.org/datasets/movielens/)数据集完成电影推荐的任务。\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "推荐系统在日常的网络应用中无处不在，比如网上购物、网上买书、新闻app、社交网络、音乐网站、电影网站等等等等，有人的地方就有推荐。根据个人的喜好，相同喜好人群的习惯等信息进行个性化的内容推荐。比如打开新闻类的app，因为有了个性化的内容，每个人看到的新闻首页都是不一样的。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这当然是很有用的，在信息爆炸的今天，获取信息的途径和方式多种多样，人们花费时间最多的不再是去哪获取信息，而是要在众多的信息中寻找自己感兴趣的，这就是信息超载问题。为了解决这个问题，推荐系统应运而生。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "协同过滤是推荐系统应用较广泛的技术，该方法搜集用户的历史记录、个人喜好等信息，计算与其他用户的相似度，利用相似用户的评价来预测目标用户对特定项目的喜好程度。优点是会给用户推荐未浏览过的项目，缺点呢，对于新用户来说，没有任何与商品的交互记录和个人喜好等信息，存在冷启动问题，导致模型无法找到相似的用户或商品。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "为了解决冷启动的问题，通常的做法是对于刚注册的用户，要求用户先选择自己感兴趣的话题、群组、商品、性格、喜欢的音乐类型等信息，比如豆瓣FM：\n",
    "<img src=\"assets/IMG_6242_300.PNG\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 下载数据集\n",
    "运行下面代码把[`数据集`](http://files.grouplens.org/datasets/movielens/ml-1m.zip)下载下来"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from sklearn.model_selection import train_test_split\n",
    "import numpy as np\n",
    "from collections import Counter\n",
    "import tensorflow as tf\n",
    "\n",
    "import os\n",
    "import pickle\n",
    "import re\n",
    "from tensorflow.python.ops import math_ops"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from urllib.request import urlretrieve\n",
    "from os.path import isfile, isdir\n",
    "from tqdm import tqdm\n",
    "import zipfile\n",
    "import hashlib\n",
    "\n",
    "def _unzip(save_path, _, database_name, data_path):\n",
    "    \"\"\"\n",
    "    Unzip wrapper with the same interface as _ungzip\n",
    "    :param save_path: The path of the gzip files\n",
    "    :param database_name: Name of database\n",
    "    :param data_path: Path to extract to\n",
    "    :param _: HACK - Used to have to same interface as _ungzip\n",
    "    \"\"\"\n",
    "    print('Extracting {}...'.format(database_name))\n",
    "    with zipfile.ZipFile(save_path) as zf:\n",
    "        zf.extractall(data_path)\n",
    "\n",
    "def download_extract(database_name, data_path):\n",
    "    \"\"\"\n",
    "    Download and extract database\n",
    "    :param database_name: Database name\n",
    "    \"\"\"\n",
    "    DATASET_ML1M = 'ml-1m'\n",
    "\n",
    "    if database_name == DATASET_ML1M:\n",
    "        url = 'http://files.grouplens.org/datasets/movielens/ml-1m.zip'\n",
    "        hash_code = 'c4d9eecfca2ab87c1945afe126590906'\n",
    "        extract_path = os.path.join(data_path, 'ml-1m')\n",
    "        save_path = os.path.join(data_path, 'ml-1m.zip')\n",
    "        extract_fn = _unzip\n",
    "\n",
    "    if os.path.exists(extract_path):\n",
    "        print('Found {} Data'.format(database_name))\n",
    "        return\n",
    "\n",
    "    if not os.path.exists(data_path):\n",
    "        os.makedirs(data_path)\n",
    "\n",
    "    if not os.path.exists(save_path):\n",
    "        with DLProgress(unit='B', unit_scale=True, miniters=1, desc='Downloading {}'.format(database_name)) as pbar:\n",
    "            urlretrieve(\n",
    "                url,\n",
    "                save_path,\n",
    "                pbar.hook)\n",
    "\n",
    "    assert hashlib.md5(open(save_path, 'rb').read()).hexdigest() == hash_code, \\\n",
    "        '{} file is corrupted.  Remove the file and try again.'.format(save_path)\n",
    "\n",
    "    os.makedirs(extract_path)\n",
    "    try:\n",
    "        extract_fn(save_path, extract_path, database_name, data_path)\n",
    "    except Exception as err:\n",
    "        shutil.rmtree(extract_path)  # Remove extraction folder if there is an error\n",
    "        raise err\n",
    "\n",
    "    print('Done.')\n",
    "    # Remove compressed data\n",
    "#     os.remove(save_path)\n",
    "\n",
    "class DLProgress(tqdm):\n",
    "    \"\"\"\n",
    "    Handle Progress Bar while Downloading\n",
    "    \"\"\"\n",
    "    last_block = 0\n",
    "\n",
    "    def hook(self, block_num=1, block_size=1, total_size=None):\n",
    "        \"\"\"\n",
    "        A hook function that will be called once on establishment of the network connection and\n",
    "        once after each block read thereafter.\n",
    "        :param block_num: A count of blocks transferred so far\n",
    "        :param block_size: Block size in bytes\n",
    "        :param total_size: The total size of the file. This may be -1 on older FTP servers which do not return\n",
    "                            a file size in response to a retrieval request.\n",
    "        \"\"\"\n",
    "        self.total = total_size\n",
    "        self.update((block_num - self.last_block) * block_size)\n",
    "        self.last_block = block_num"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Downloading ml-1m: 5.92MB [00:13, 446KB/s]                             \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting ml-1m...\n",
      "Done.\n"
     ]
    }
   ],
   "source": [
    "data_dir = './'\n",
    "download_extract('ml-1m', data_dir)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 先来看看数据"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "本项目使用的是MovieLens 1M 数据集，包含6000个用户在近4000部电影上的1亿条评论。\n",
    "\n",
    "数据集分为三个文件：用户数据users.dat，电影数据movies.dat和评分数据ratings.dat。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 用户数据\n",
    "分别有用户ID、性别、年龄、职业ID和邮编等字段。\n",
    "\n",
    "数据中的格式：UserID::Gender::Age::Occupation::Zip-code\n",
    "\n",
    "- Gender is denoted by a \"M\" for male and \"F\" for female\n",
    "- Age is chosen from the following ranges:\n",
    "\n",
    "\t*  1:  \"Under 18\"\n",
    "\t* 18:  \"18-24\"\n",
    "\t* 25:  \"25-34\"\n",
    "\t* 35:  \"35-44\"\n",
    "\t* 45:  \"45-49\"\n",
    "\t* 50:  \"50-55\"\n",
    "\t* 56:  \"56+\"\n",
    "\n",
    "- Occupation is chosen from the following choices:\n",
    "\n",
    "\t*  0:  \"other\" or not specified\n",
    "\t*  1:  \"academic/educator\"\n",
    "\t*  2:  \"artist\"\n",
    "\t*  3:  \"clerical/admin\"\n",
    "\t*  4:  \"college/grad student\"\n",
    "\t*  5:  \"customer service\"\n",
    "\t*  6:  \"doctor/health care\"\n",
    "\t*  7:  \"executive/managerial\"\n",
    "\t*  8:  \"farmer\"\n",
    "\t*  9:  \"homemaker\"\n",
    "\t* 10:  \"K-12 student\"\n",
    "\t* 11:  \"lawyer\"\n",
    "\t* 12:  \"programmer\"\n",
    "\t* 13:  \"retired\"\n",
    "\t* 14:  \"sales/marketing\"\n",
    "\t* 15:  \"scientist\"\n",
    "\t* 16:  \"self-employed\"\n",
    "\t* 17:  \"technician/engineer\"\n",
    "\t* 18:  \"tradesman/craftsman\"\n",
    "\t* 19:  \"unemployed\"\n",
    "\t* 20:  \"writer\"\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>UserID</th>\n",
       "      <th>Gender</th>\n",
       "      <th>Age</th>\n",
       "      <th>OccupationID</th>\n",
       "      <th>Zip-code</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>F</td>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "      <td>48067</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>M</td>\n",
       "      <td>56</td>\n",
       "      <td>16</td>\n",
       "      <td>70072</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>M</td>\n",
       "      <td>25</td>\n",
       "      <td>15</td>\n",
       "      <td>55117</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>M</td>\n",
       "      <td>45</td>\n",
       "      <td>7</td>\n",
       "      <td>02460</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>M</td>\n",
       "      <td>25</td>\n",
       "      <td>20</td>\n",
       "      <td>55455</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   UserID Gender  Age  OccupationID Zip-code\n",
       "0       1      F    1            10    48067\n",
       "1       2      M   56            16    70072\n",
       "2       3      M   25            15    55117\n",
       "3       4      M   45             7    02460\n",
       "4       5      M   25            20    55455"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "users_title = ['UserID', 'Gender', 'Age', 'OccupationID', 'Zip-code']\n",
    "users = pd.read_table('./ml-1m/users.dat', sep='::', header=None, names=users_title, engine = 'python')\n",
    "users.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以看出UserID、Gender、Age和Occupation都是类别字段，其中邮编字段是我们不使用的。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 电影数据\n",
    "分别有电影ID、电影名和电影风格等字段。\n",
    "\n",
    "数据中的格式：MovieID::Title::Genres\n",
    "\n",
    "- Titles are identical to titles provided by the IMDB (including\n",
    "year of release)\n",
    "- Genres are pipe-separated and are selected from the following genres:\n",
    "\n",
    "\t* Action\n",
    "\t* Adventure\n",
    "\t* Animation\n",
    "\t* Children's\n",
    "\t* Comedy\n",
    "\t* Crime\n",
    "\t* Documentary\n",
    "\t* Drama\n",
    "\t* Fantasy\n",
    "\t* Film-Noir\n",
    "\t* Horror\n",
    "\t* Musical\n",
    "\t* Mystery\n",
    "\t* Romance\n",
    "\t* Sci-Fi\n",
    "\t* Thriller\n",
    "\t* War\n",
    "\t* Western\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MovieID</th>\n",
       "      <th>Title</th>\n",
       "      <th>Genres</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Toy Story (1995)</td>\n",
       "      <td>Animation|Children's|Comedy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>Jumanji (1995)</td>\n",
       "      <td>Adventure|Children's|Fantasy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>Grumpier Old Men (1995)</td>\n",
       "      <td>Comedy|Romance</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>Waiting to Exhale (1995)</td>\n",
       "      <td>Comedy|Drama</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>Father of the Bride Part II (1995)</td>\n",
       "      <td>Comedy</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   MovieID                               Title                        Genres\n",
       "0        1                    Toy Story (1995)   Animation|Children's|Comedy\n",
       "1        2                      Jumanji (1995)  Adventure|Children's|Fantasy\n",
       "2        3             Grumpier Old Men (1995)                Comedy|Romance\n",
       "3        4            Waiting to Exhale (1995)                  Comedy|Drama\n",
       "4        5  Father of the Bride Part II (1995)                        Comedy"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movies_title = ['MovieID', 'Title', 'Genres']\n",
    "movies = pd.read_table('./ml-1m/movies.dat', sep='::', header=None, names=movies_title, engine = 'python')\n",
    "movies.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "MovieID是类别字段，Title是文本，Genres也是类别字段"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 评分数据\n",
    "分别有用户ID、电影ID、评分和时间戳等字段。\n",
    "\n",
    "数据中的格式：UserID::MovieID::Rating::Timestamp\n",
    "\n",
    "- UserIDs range between 1 and 6040 \n",
    "- MovieIDs range between 1 and 3952\n",
    "- Ratings are made on a 5-star scale (whole-star ratings only)\n",
    "- Timestamp is represented in seconds since the epoch as returned by time(2)\n",
    "- Each user has at least 20 ratings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>UserID</th>\n",
       "      <th>MovieID</th>\n",
       "      <th>Rating</th>\n",
       "      <th>timestamps</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1193</td>\n",
       "      <td>5</td>\n",
       "      <td>978300760</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>661</td>\n",
       "      <td>3</td>\n",
       "      <td>978302109</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>914</td>\n",
       "      <td>3</td>\n",
       "      <td>978301968</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>3408</td>\n",
       "      <td>4</td>\n",
       "      <td>978300275</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>2355</td>\n",
       "      <td>5</td>\n",
       "      <td>978824291</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   UserID  MovieID  Rating  timestamps\n",
       "0       1     1193       5   978300760\n",
       "1       1      661       3   978302109\n",
       "2       1      914       3   978301968\n",
       "3       1     3408       4   978300275\n",
       "4       1     2355       5   978824291"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ratings_title = ['UserID','MovieID', 'Rating', 'timestamps']\n",
    "ratings = pd.read_table('./ml-1m/ratings.dat', sep='::', header=None, names=ratings_title, engine = 'python')\n",
    "ratings.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "评分字段Rating就是我们要学习的targets，时间戳字段我们不使用。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 来说说数据预处理"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- UserID、Occupation和MovieID不用变。\n",
    "- Gender字段：需要将‘F’和‘M’转换成0和1。\n",
    "- Age字段：要转成7个连续数字0~6。\n",
    "- Genres字段：是分类字段，要转成数字。首先将Genres中的类别转成字符串到数字的字典，然后再将每个电影的Genres字段转成数字列表，因为有些电影是多个Genres的组合。\n",
    "- Title字段：处理方式跟Genres字段一样，首先创建文本到数字的字典，然后将Title中的描述转成数字的列表。另外Title中的年份也需要去掉。\n",
    "- Genres和Title字段需要将长度统一，这样在神经网络中方便处理。空白部分用‘< PAD >’对应的数字填充。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 实现数据预处理"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "def load_data():\n",
    "    \"\"\"\n",
    "    Load Dataset from File\n",
    "    \"\"\"\n",
    "    #读取User数据\n",
    "    users_title = ['UserID', 'Gender', 'Age', 'JobID', 'Zip-code']\n",
    "    users = pd.read_table('./ml-1m/users.dat', sep='::', header=None, names=users_title, engine = 'python')\n",
    "    users = users.filter(regex='UserID|Gender|Age|JobID')\n",
    "    users_orig = users.values\n",
    "    #改变User数据中性别和年龄\n",
    "    gender_map = {'F':0, 'M':1}\n",
    "    users['Gender'] = users['Gender'].map(gender_map)\n",
    "\n",
    "    age_map = {val:ii for ii,val in enumerate(set(users['Age']))}\n",
    "    users['Age'] = users['Age'].map(age_map)\n",
    "\n",
    "    #读取Movie数据集\n",
    "    movies_title = ['MovieID', 'Title', 'Genres']\n",
    "    movies = pd.read_table('./ml-1m/movies.dat', sep='::', header=None, names=movies_title, engine = 'python')\n",
    "    movies_orig = movies.values\n",
    "    #将Title中的年份去掉\n",
    "    pattern = re.compile(r'^(.*)\\((\\d+)\\)$')\n",
    "\n",
    "    title_map = {val:pattern.match(val).group(1) for ii,val in enumerate(set(movies['Title']))}\n",
    "    movies['Title'] = movies['Title'].map(title_map)\n",
    "\n",
    "    #电影类型转数字字典\n",
    "    genres_set = set()\n",
    "    for val in movies['Genres'].str.split('|'):\n",
    "        genres_set.update(val)\n",
    "\n",
    "    genres_set.add('<PAD>')\n",
    "    genres2int = {val:ii for ii, val in enumerate(genres_set)}\n",
    "\n",
    "    #将电影类型转成等长数字列表，长度是18\n",
    "    genres_map = {val:[genres2int[row] for row in val.split('|')] for ii,val in enumerate(set(movies['Genres']))}\n",
    "\n",
    "    for key in genres_map:\n",
    "        for cnt in range(max(genres2int.values()) - len(genres_map[key])):\n",
    "            genres_map[key].insert(len(genres_map[key]) + cnt,genres2int['<PAD>'])\n",
    "    \n",
    "    movies['Genres'] = movies['Genres'].map(genres_map)\n",
    "\n",
    "    #电影Title转数字字典\n",
    "    title_set = set()\n",
    "    for val in movies['Title'].str.split():\n",
    "        title_set.update(val)\n",
    "    \n",
    "    title_set.add('<PAD>')\n",
    "    title2int = {val:ii for ii, val in enumerate(title_set)}\n",
    "\n",
    "    #将电影Title转成等长数字列表，长度是15\n",
    "    title_count = 15\n",
    "    title_map = {val:[title2int[row] for row in val.split()] for ii,val in enumerate(set(movies['Title']))}\n",
    "    \n",
    "    for key in title_map:\n",
    "        for cnt in range(title_count - len(title_map[key])):\n",
    "            title_map[key].insert(len(title_map[key]) + cnt,title2int['<PAD>'])\n",
    "    \n",
    "    movies['Title'] = movies['Title'].map(title_map)\n",
    "\n",
    "    #读取评分数据集\n",
    "    ratings_title = ['UserID','MovieID', 'ratings', 'timestamps']\n",
    "    ratings = pd.read_table('./ml-1m/ratings.dat', sep='::', header=None, names=ratings_title, engine = 'python')\n",
    "    ratings = ratings.filter(regex='UserID|MovieID|ratings')\n",
    "\n",
    "    #合并三个表\n",
    "    data = pd.merge(pd.merge(ratings, users), movies)\n",
    "    \n",
    "    #将数据分成X和y两张表\n",
    "    target_fields = ['ratings']\n",
    "    features_pd, targets_pd = data.drop(target_fields, axis=1), data[target_fields]\n",
    "    \n",
    "    features = features_pd.values\n",
    "    targets_values = targets_pd.values\n",
    "    \n",
    "    return title_count, title_set, genres2int, features, targets_values, ratings, users, movies, data, movies_orig, users_orig"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 加载数据并保存到本地"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- title_count：Title字段的长度（15）\n",
    "- title_set：Title文本的集合\n",
    "- genres2int：电影类型转数字的字典\n",
    "- features：是输入X\n",
    "- targets_values：是学习目标y\n",
    "- ratings：评分数据集的Pandas对象\n",
    "- users：用户数据集的Pandas对象\n",
    "- movies：电影数据的Pandas对象\n",
    "- data：三个数据集组合在一起的Pandas对象\n",
    "- movies_orig：没有做数据处理的原始电影数据\n",
    "- users_orig：没有做数据处理的原始用户数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "title_count, title_set, genres2int, features, targets_values, ratings, users, movies, data, movies_orig, users_orig = load_data()\n",
    "\n",
    "pickle.dump((title_count, title_set, genres2int, features, targets_values, ratings, users, movies, data, movies_orig, users_orig), open('preprocess.p', 'wb'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 预处理后的数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>UserID</th>\n",
       "      <th>Gender</th>\n",
       "      <th>Age</th>\n",
       "      <th>JobID</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   UserID  Gender  Age  JobID\n",
       "0       1       0    0     10\n",
       "1       2       1    5     16\n",
       "2       3       1    6     15\n",
       "3       4       1    2      7\n",
       "4       5       1    6     20"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "users.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MovieID</th>\n",
       "      <th>Title</th>\n",
       "      <th>Genres</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>[3001, 5100, 275, 275, 275, 275, 275, 275, 275...</td>\n",
       "      <td>[3, 6, 2, 17, 17, 17, 17, 17, 17, 17, 17, 17, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>[2280, 275, 275, 275, 275, 275, 275, 275, 275,...</td>\n",
       "      <td>[16, 6, 14, 17, 17, 17, 17, 17, 17, 17, 17, 17...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>[4339, 3338, 348, 275, 275, 275, 275, 275, 275...</td>\n",
       "      <td>[2, 13, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>[4507, 4093, 596, 275, 275, 275, 275, 275, 275...</td>\n",
       "      <td>[2, 4, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>[2123, 4479, 2698, 2221, 4495, 3997, 275, 275,...</td>\n",
       "      <td>[2, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   MovieID                                              Title  \\\n",
       "0        1  [3001, 5100, 275, 275, 275, 275, 275, 275, 275...   \n",
       "1        2  [2280, 275, 275, 275, 275, 275, 275, 275, 275,...   \n",
       "2        3  [4339, 3338, 348, 275, 275, 275, 275, 275, 275...   \n",
       "3        4  [4507, 4093, 596, 275, 275, 275, 275, 275, 275...   \n",
       "4        5  [2123, 4479, 2698, 2221, 4495, 3997, 275, 275,...   \n",
       "\n",
       "                                              Genres  \n",
       "0  [3, 6, 2, 17, 17, 17, 17, 17, 17, 17, 17, 17, ...  \n",
       "1  [16, 6, 14, 17, 17, 17, 17, 17, 17, 17, 17, 17...  \n",
       "2  [2, 13, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17...  \n",
       "3  [2, 4, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17,...  \n",
       "4  [2, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17...  "
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movies.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1,\n",
       "       list([3001, 5100, 275, 275, 275, 275, 275, 275, 275, 275, 275, 275, 275, 275, 275]),\n",
       "       list([3, 6, 2, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17])], dtype=object)"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movies.values[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 从本地读取数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "title_count, title_set, genres2int, features, targets_values, ratings, users, movies, data, movies_orig, users_orig = pickle.load(open('preprocess.p', mode='rb'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 模型设计"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"assets/model.001.jpeg\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "通过研究数据集中的字段类型，我们发现有一些是类别字段，通常的处理是将这些字段转成one hot编码，但是像UserID、MovieID这样的字段就会变成非常的稀疏，输入的维度急剧膨胀，这是我们不愿意见到的，毕竟我这小笔记本不像大厂动辄能处理数以亿计维度的输入：）\n",
    "\n",
    "所以在预处理数据时将这些字段转成了数字，我们用这个数字当做嵌入矩阵的索引，在网络的第一层使用了嵌入层，维度是（N，32）和（N，16）。\n",
    "\n",
    "电影类型的处理要多一步，有时一个电影有多个电影类型，这样从嵌入矩阵索引出来是一个（n，32）的矩阵，因为有多个类型嘛，我们要将这个矩阵求和，变成（1，32）的向量。\n",
    "\n",
    "电影名的处理比较特殊，没有使用循环神经网络，而是用了文本卷积网络，下文会进行说明。\n",
    "\n",
    "从嵌入层索引出特征以后，将各特征传入全连接层，将输出再次传入全连接层，最终分别得到（1，200）的用户特征和电影特征两个特征向量。\n",
    "\n",
    "我们的目的就是要训练出用户特征和电影特征，在实现推荐功能时使用。得到这两个特征以后，就可以选择任意的方式来拟合评分了。我使用了两种方式，一个是上图中画出的将两个特征做向量乘法，将结果与真实评分做回归，采用MSE优化损失。因为本质上这是一个回归问题，另一种方式是，将两个特征作为输入，再次传入全连接层，输出一个值，将输出值回归到真实评分，采用MSE优化损失。\n",
    "\n",
    "实际上第二个方式的MSE loss在0.8附近，第一个方式在1附近，5次迭代的结果。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 文本卷积网络\n",
    "网络看起来像下面这样"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"assets/text_cnn.png\"/>\n",
    "图片来自Kim Yoon的论文：[`Convolutional Neural Networks for Sentence Classification`](https://arxiv.org/abs/1408.5882)\n",
    "\n",
    "将卷积神经网络用于文本的文章建议你阅读[`Understanding Convolutional Neural Networks for NLP`](http://www.wildml.com/2015/11/understanding-convolutional-neural-networks-for-nlp/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "网络的第一层是词嵌入层，由每一个单词的嵌入向量组成的嵌入矩阵。下一层使用多个不同尺寸（窗口大小）的卷积核在嵌入矩阵上做卷积，窗口大小指的是每次卷积覆盖几个单词。这里跟对图像做卷积不太一样，图像的卷积通常用2x2、3x3、5x5之类的尺寸，而文本卷积要覆盖整个单词的嵌入向量，所以尺寸是（单词数，向量维度），比如每次滑动3个，4个或者5个单词。第三层网络是max pooling得到一个长向量，最后使用dropout做正则化，最终得到了电影Title的特征。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 辅助函数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import os\n",
    "import pickle\n",
    "\n",
    "def save_params(params):\n",
    "    \"\"\"\n",
    "    Save parameters to file\n",
    "    \"\"\"\n",
    "    pickle.dump(params, open('params.p', 'wb'))\n",
    "\n",
    "\n",
    "def load_params():\n",
    "    \"\"\"\n",
    "    Load parameters from file\n",
    "    \"\"\"\n",
    "    return pickle.load(open('params.p', mode='rb'))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 编码实现"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "#嵌入矩阵的维度\n",
    "embed_dim = 32\n",
    "#用户ID个数\n",
    "uid_max = max(features.take(0,1)) + 1 # 6040\n",
    "#性别个数\n",
    "gender_max = max(features.take(2,1)) + 1 # 1 + 1 = 2\n",
    "#年龄类别个数\n",
    "age_max = max(features.take(3,1)) + 1 # 6 + 1 = 7\n",
    "#职业个数\n",
    "job_max = max(features.take(4,1)) + 1# 20 + 1 = 21\n",
    "\n",
    "#电影ID个数\n",
    "movie_id_max = max(features.take(1,1)) + 1 # 3952\n",
    "#电影类型个数\n",
    "movie_categories_max = max(genres2int.values()) + 1 # 18 + 1 = 19\n",
    "#电影名单词个数\n",
    "movie_title_max = len(title_set) # 5216\n",
    "\n",
    "#对电影类型嵌入向量做加和操作的标志，考虑过使用mean做平均，但是没实现mean\n",
    "combiner = \"sum\"\n",
    "\n",
    "#电影名长度\n",
    "sentences_size = title_count # = 15\n",
    "#文本卷积滑动窗口，分别滑动2, 3, 4, 5个单词\n",
    "window_sizes = {2, 3, 4, 5}\n",
    "#文本卷积核数量\n",
    "filter_num = 8\n",
    "\n",
    "#电影ID转下标的字典，数据集中电影ID跟下标不一致，比如第5行的数据电影ID不一定是5\n",
    "movieid2idx = {val[0]:i for i, val in enumerate(movies.values)}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 超参"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Number of Epochs\n",
    "num_epochs = 5\n",
    "# Batch Size\n",
    "batch_size = 256\n",
    "\n",
    "dropout_keep = 0.5\n",
    "# Learning Rate\n",
    "learning_rate = 0.0001\n",
    "# Show stats for every n number of batches\n",
    "show_every_n_batches = 20\n",
    "\n",
    "save_dir = './save'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 输入"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "定义输入的占位符"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_inputs():\n",
    "    uid = tf.placeholder(tf.int32, [None, 1], name=\"uid\")\n",
    "    user_gender = tf.placeholder(tf.int32, [None, 1], name=\"user_gender\")\n",
    "    user_age = tf.placeholder(tf.int32, [None, 1], name=\"user_age\")\n",
    "    user_job = tf.placeholder(tf.int32, [None, 1], name=\"user_job\")\n",
    "    \n",
    "    movie_id = tf.placeholder(tf.int32, [None, 1], name=\"movie_id\")\n",
    "    movie_categories = tf.placeholder(tf.int32, [None, 18], name=\"movie_categories\")\n",
    "    movie_titles = tf.placeholder(tf.int32, [None, 15], name=\"movie_titles\")\n",
    "    targets = tf.placeholder(tf.int32, [None, 1], name=\"targets\")\n",
    "    LearningRate = tf.placeholder(tf.float32, name = \"LearningRate\")\n",
    "    dropout_keep_prob = tf.placeholder(tf.float32, name = \"dropout_keep_prob\")\n",
    "    return uid, user_gender, user_age, user_job, movie_id, movie_categories, movie_titles, targets, LearningRate, dropout_keep_prob"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 构建神经网络"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 定义User的嵌入矩阵"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_user_embedding(uid, user_gender, user_age, user_job):\n",
    "    with tf.name_scope(\"user_embedding\"):\n",
    "        uid_embed_matrix = tf.Variable(tf.random_uniform([uid_max, embed_dim], -1, 1), name = \"uid_embed_matrix\")\n",
    "        uid_embed_layer = tf.nn.embedding_lookup(uid_embed_matrix, uid, name = \"uid_embed_layer\")\n",
    "    \n",
    "        gender_embed_matrix = tf.Variable(tf.random_uniform([gender_max, embed_dim // 2], -1, 1), name= \"gender_embed_matrix\")\n",
    "        gender_embed_layer = tf.nn.embedding_lookup(gender_embed_matrix, user_gender, name = \"gender_embed_layer\")\n",
    "        \n",
    "        age_embed_matrix = tf.Variable(tf.random_uniform([age_max, embed_dim // 2], -1, 1), name=\"age_embed_matrix\")\n",
    "        age_embed_layer = tf.nn.embedding_lookup(age_embed_matrix, user_age, name=\"age_embed_layer\")\n",
    "        \n",
    "        job_embed_matrix = tf.Variable(tf.random_uniform([job_max, embed_dim // 2], -1, 1), name = \"job_embed_matrix\")\n",
    "        job_embed_layer = tf.nn.embedding_lookup(job_embed_matrix, user_job, name = \"job_embed_layer\")\n",
    "    return uid_embed_layer, gender_embed_layer, age_embed_layer, job_embed_layer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 将User的嵌入矩阵一起全连接生成User的特征"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_user_feature_layer(uid_embed_layer, gender_embed_layer, age_embed_layer, job_embed_layer):\n",
    "    with tf.name_scope(\"user_fc\"):\n",
    "        #第一层全连接\n",
    "        uid_fc_layer = tf.layers.dense(uid_embed_layer, embed_dim, name = \"uid_fc_layer\", activation=tf.nn.relu)\n",
    "        gender_fc_layer = tf.layers.dense(gender_embed_layer, embed_dim, name = \"gender_fc_layer\", activation=tf.nn.relu)\n",
    "        age_fc_layer = tf.layers.dense(age_embed_layer, embed_dim, name =\"age_fc_layer\", activation=tf.nn.relu)\n",
    "        job_fc_layer = tf.layers.dense(job_embed_layer, embed_dim, name = \"job_fc_layer\", activation=tf.nn.relu)\n",
    "        \n",
    "        #第二层全连接\n",
    "        user_combine_layer = tf.concat([uid_fc_layer, gender_fc_layer, age_fc_layer, job_fc_layer], 2)  #(?, 1, 128)\n",
    "        user_combine_layer = tf.contrib.layers.fully_connected(user_combine_layer, 200, tf.tanh)  #(?, 1, 200)\n",
    "    \n",
    "        user_combine_layer_flat = tf.reshape(user_combine_layer, [-1, 200])\n",
    "    return user_combine_layer, user_combine_layer_flat"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 定义Movie ID的嵌入矩阵"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_movie_id_embed_layer(movie_id):\n",
    "    with tf.name_scope(\"movie_embedding\"):\n",
    "        movie_id_embed_matrix = tf.Variable(tf.random_uniform([movie_id_max, embed_dim], -1, 1), name = \"movie_id_embed_matrix\")\n",
    "        movie_id_embed_layer = tf.nn.embedding_lookup(movie_id_embed_matrix, movie_id, name = \"movie_id_embed_layer\")\n",
    "    return movie_id_embed_layer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 对电影类型的多个嵌入向量做加和"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_movie_categories_layers(movie_categories):\n",
    "    with tf.name_scope(\"movie_categories_layers\"):\n",
    "        movie_categories_embed_matrix = tf.Variable(tf.random_uniform([movie_categories_max, embed_dim], -1, 1), name = \"movie_categories_embed_matrix\")\n",
    "        movie_categories_embed_layer = tf.nn.embedding_lookup(movie_categories_embed_matrix, movie_categories, name = \"movie_categories_embed_layer\")\n",
    "        if combiner == \"sum\":\n",
    "            movie_categories_embed_layer = tf.reduce_sum(movie_categories_embed_layer, axis=1, keep_dims=True)\n",
    "    #     elif combiner == \"mean\":\n",
    "\n",
    "    return movie_categories_embed_layer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Movie Title的文本卷积网络实现"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_movie_cnn_layer(movie_titles):\n",
    "    #从嵌入矩阵中得到电影名对应的各个单词的嵌入向量\n",
    "    with tf.name_scope(\"movie_embedding\"):\n",
    "        movie_title_embed_matrix = tf.Variable(tf.random_uniform([movie_title_max, embed_dim], -1, 1), name = \"movie_title_embed_matrix\")\n",
    "        movie_title_embed_layer = tf.nn.embedding_lookup(movie_title_embed_matrix, movie_titles, name = \"movie_title_embed_layer\")\n",
    "        movie_title_embed_layer_expand = tf.expand_dims(movie_title_embed_layer, -1)\n",
    "    \n",
    "    #对文本嵌入层使用不同尺寸的卷积核做卷积和最大池化\n",
    "    pool_layer_lst = []\n",
    "    for window_size in window_sizes:\n",
    "        with tf.name_scope(\"movie_txt_conv_maxpool_{}\".format(window_size)):\n",
    "            filter_weights = tf.Variable(tf.truncated_normal([window_size, embed_dim, 1, filter_num],stddev=0.1),name = \"filter_weights\")\n",
    "            filter_bias = tf.Variable(tf.constant(0.1, shape=[filter_num]), name=\"filter_bias\")\n",
    "            \n",
    "            conv_layer = tf.nn.conv2d(movie_title_embed_layer_expand, filter_weights, [1,1,1,1], padding=\"VALID\", name=\"conv_layer\")\n",
    "            relu_layer = tf.nn.relu(tf.nn.bias_add(conv_layer,filter_bias), name =\"relu_layer\")\n",
    "            \n",
    "            maxpool_layer = tf.nn.max_pool(relu_layer, [1,sentences_size - window_size + 1 ,1,1], [1,1,1,1], padding=\"VALID\", name=\"maxpool_layer\")\n",
    "            pool_layer_lst.append(maxpool_layer)\n",
    "\n",
    "    #Dropout层\n",
    "    with tf.name_scope(\"pool_dropout\"):\n",
    "        pool_layer = tf.concat(pool_layer_lst, 3, name =\"pool_layer\")\n",
    "        max_num = len(window_sizes) * filter_num\n",
    "        pool_layer_flat = tf.reshape(pool_layer , [-1, 1, max_num], name = \"pool_layer_flat\")\n",
    "    \n",
    "        dropout_layer = tf.nn.dropout(pool_layer_flat, dropout_keep_prob, name = \"dropout_layer\")\n",
    "    return pool_layer_flat, dropout_layer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 将Movie的各个层一起做全连接"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_movie_feature_layer(movie_id_embed_layer, movie_categories_embed_layer, dropout_layer):\n",
    "    with tf.name_scope(\"movie_fc\"):\n",
    "        #第一层全连接\n",
    "        movie_id_fc_layer = tf.layers.dense(movie_id_embed_layer, embed_dim, name = \"movie_id_fc_layer\", activation=tf.nn.relu)\n",
    "        movie_categories_fc_layer = tf.layers.dense(movie_categories_embed_layer, embed_dim, name = \"movie_categories_fc_layer\", activation=tf.nn.relu)\n",
    "    \n",
    "        #第二层全连接\n",
    "        movie_combine_layer = tf.concat([movie_id_fc_layer, movie_categories_fc_layer, dropout_layer], 2)  #(?, 1, 96)\n",
    "        movie_combine_layer = tf.contrib.layers.fully_connected(movie_combine_layer, 200, tf.tanh)  #(?, 1, 200)\n",
    "    \n",
    "        movie_combine_layer_flat = tf.reshape(movie_combine_layer, [-1, 200])\n",
    "    return movie_combine_layer, movie_combine_layer_flat"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 构建计算图"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.reset_default_graph()\n",
    "train_graph = tf.Graph()\n",
    "with train_graph.as_default():\n",
    "    #获取输入占位符\n",
    "    uid, user_gender, user_age, user_job, movie_id, movie_categories, movie_titles, targets, lr, dropout_keep_prob = get_inputs()\n",
    "    #获取User的4个嵌入向量\n",
    "    uid_embed_layer, gender_embed_layer, age_embed_layer, job_embed_layer = get_user_embedding(uid, user_gender, user_age, user_job)\n",
    "    #得到用户特征\n",
    "    user_combine_layer, user_combine_layer_flat = get_user_feature_layer(uid_embed_layer, gender_embed_layer, age_embed_layer, job_embed_layer)\n",
    "    #获取电影ID的嵌入向量\n",
    "    movie_id_embed_layer = get_movie_id_embed_layer(movie_id)\n",
    "    #获取电影类型的嵌入向量\n",
    "    movie_categories_embed_layer = get_movie_categories_layers(movie_categories)\n",
    "    #获取电影名的特征向量\n",
    "    pool_layer_flat, dropout_layer = get_movie_cnn_layer(movie_titles)\n",
    "    #得到电影特征\n",
    "    movie_combine_layer, movie_combine_layer_flat = get_movie_feature_layer(movie_id_embed_layer, \n",
    "                                                                                movie_categories_embed_layer, \n",
    "                                                                                dropout_layer)\n",
    "    #计算出评分，要注意两个不同的方案，inference的名字（name值）是不一样的，后面做推荐时要根据name取得tensor\n",
    "    with tf.name_scope(\"inference\"):\n",
    "        #将用户特征和电影特征作为输入，经过全连接，输出一个值的方案\n",
    "#         inference_layer = tf.concat([user_combine_layer_flat, movie_combine_layer_flat], 1)  #(?, 200)\n",
    "#         inference = tf.layers.dense(inference_layer, 1,\n",
    "#                                     kernel_initializer=tf.truncated_normal_initializer(stddev=0.01), \n",
    "#                                     kernel_regularizer=tf.nn.l2_loss, name=\"inference\")\n",
    "        #简单的将用户特征和电影特征做矩阵乘法得到一个预测评分\n",
    "#        inference = tf.matmul(user_combine_layer_flat, tf.transpose(movie_combine_layer_flat))\n",
    "        inference = tf.reduce_sum(user_combine_layer_flat * movie_combine_layer_flat, axis=1)\n",
    "        inference = tf.expand_dims(inference, axis=1)\n",
    "\n",
    "    with tf.name_scope(\"loss\"):\n",
    "        # MSE损失，将计算值回归到评分\n",
    "        cost = tf.losses.mean_squared_error(targets, inference )\n",
    "        loss = tf.reduce_mean(cost)\n",
    "    # 优化损失 \n",
    "#     train_op = tf.train.AdamOptimizer(lr).minimize(loss)  #cost\n",
    "    global_step = tf.Variable(0, name=\"global_step\", trainable=False)\n",
    "    optimizer = tf.train.AdamOptimizer(lr)\n",
    "    gradients = optimizer.compute_gradients(loss)  #cost\n",
    "    train_op = optimizer.apply_gradients(gradients, global_step=global_step)\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor 'inference/MatMul:0' shape=(?, ?) dtype=float32>"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "inference"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 取得batch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_batches(Xs, ys, batch_size):\n",
    "    for start in range(0, len(Xs), batch_size):\n",
    "        end = min(start + batch_size, len(Xs))\n",
    "        yield Xs[start:end], ys[start:end]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 训练网络"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing to /Users/chengstone/Downloads/cn-deep-learning-master/recommender/runs/1513402825\n",
      "\n",
      "2017-12-16T13:40:36.409025: Epoch   0 Batch    0/3125   train_loss = 12.309\n",
      "2017-12-16T13:40:38.337879: Epoch   0 Batch   20/3125   train_loss = 3.581\n",
      "2017-12-16T13:40:40.269757: Epoch   0 Batch   40/3125   train_loss = 2.589\n",
      "2017-12-16T13:40:42.132878: Epoch   0 Batch   60/3125   train_loss = 1.856\n",
      "2017-12-16T13:40:44.032095: Epoch   0 Batch   80/3125   train_loss = 1.856\n",
      "2017-12-16T13:40:45.901675: Epoch   0 Batch  100/3125   train_loss = 1.629\n",
      "2017-12-16T13:40:48.056503: Epoch   0 Batch  120/3125   train_loss = 1.771\n",
      "2017-12-16T13:40:49.929440: Epoch   0 Batch  140/3125   train_loss = 1.629\n",
      "2017-12-16T13:40:51.808643: Epoch   0 Batch  160/3125   train_loss = 1.397\n",
      "2017-12-16T13:40:53.692910: Epoch   0 Batch  180/3125   train_loss = 1.483\n",
      "2017-12-16T13:40:55.512025: Epoch   0 Batch  200/3125   train_loss = 1.701\n",
      "2017-12-16T13:40:57.345646: Epoch   0 Batch  220/3125   train_loss = 1.508\n",
      "2017-12-16T13:40:59.200821: Epoch   0 Batch  240/3125   train_loss = 1.461\n",
      "2017-12-16T13:41:01.052601: Epoch   0 Batch  260/3125   train_loss = 1.524\n",
      "2017-12-16T13:41:02.861373: Epoch   0 Batch  280/3125   train_loss = 1.659\n",
      "2017-12-16T13:41:04.760514: Epoch   0 Batch  300/3125   train_loss = 1.477\n",
      "2017-12-16T13:41:06.639394: Epoch   0 Batch  320/3125   train_loss = 1.529\n",
      "2017-12-16T13:41:08.625204: Epoch   0 Batch  340/3125   train_loss = 1.319\n",
      "2017-12-16T13:41:10.546636: Epoch   0 Batch  360/3125   train_loss = 1.407\n",
      "2017-12-16T13:41:12.519326: Epoch   0 Batch  380/3125   train_loss = 1.383\n",
      "2017-12-16T13:41:14.412138: Epoch   0 Batch  400/3125   train_loss = 1.200\n",
      "2017-12-16T13:41:16.252121: Epoch   0 Batch  420/3125   train_loss = 1.281\n",
      "2017-12-16T13:41:18.112281: Epoch   0 Batch  440/3125   train_loss = 1.385\n",
      "2017-12-16T13:41:19.953702: Epoch   0 Batch  460/3125   train_loss = 1.395\n",
      "2017-12-16T13:41:21.835520: Epoch   0 Batch  480/3125   train_loss = 1.414\n",
      "2017-12-16T13:41:23.711496: Epoch   0 Batch  500/3125   train_loss = 1.009\n",
      "2017-12-16T13:41:25.597752: Epoch   0 Batch  520/3125   train_loss = 1.386\n",
      "2017-12-16T13:41:27.429644: Epoch   0 Batch  540/3125   train_loss = 1.239\n",
      "2017-12-16T13:41:29.335982: Epoch   0 Batch  560/3125   train_loss = 1.399\n",
      "2017-12-16T13:41:31.143999: Epoch   0 Batch  580/3125   train_loss = 1.452\n",
      "2017-12-16T13:41:33.046766: Epoch   0 Batch  600/3125   train_loss = 1.414\n",
      "2017-12-16T13:41:34.964805: Epoch   0 Batch  620/3125   train_loss = 1.429\n",
      "2017-12-16T13:41:36.831790: Epoch   0 Batch  640/3125   train_loss = 1.348\n",
      "2017-12-16T13:41:38.678891: Epoch   0 Batch  660/3125   train_loss = 1.383\n",
      "2017-12-16T13:41:40.573495: Epoch   0 Batch  680/3125   train_loss = 1.210\n",
      "2017-12-16T13:41:42.443230: Epoch   0 Batch  700/3125   train_loss = 1.273\n",
      "2017-12-16T13:41:44.335420: Epoch   0 Batch  720/3125   train_loss = 1.211\n",
      "2017-12-16T13:41:46.143437: Epoch   0 Batch  740/3125   train_loss = 1.340\n",
      "2017-12-16T13:41:47.995662: Epoch   0 Batch  760/3125   train_loss = 1.393\n",
      "2017-12-16T13:41:49.903770: Epoch   0 Batch  780/3125   train_loss = 1.440\n",
      "2017-12-16T13:41:51.790669: Epoch   0 Batch  800/3125   train_loss = 1.301\n",
      "2017-12-16T13:41:53.692870: Epoch   0 Batch  820/3125   train_loss = 1.276\n",
      "2017-12-16T13:41:55.528444: Epoch   0 Batch  840/3125   train_loss = 1.248\n",
      "2017-12-16T13:41:57.395823: Epoch   0 Batch  860/3125   train_loss = 1.187\n",
      "2017-12-16T13:41:59.269651: Epoch   0 Batch  880/3125   train_loss = 1.290\n",
      "2017-12-16T13:42:01.118377: Epoch   0 Batch  900/3125   train_loss = 1.193\n",
      "2017-12-16T13:42:03.039355: Epoch   0 Batch  920/3125   train_loss = 1.384\n",
      "2017-12-16T13:42:04.893670: Epoch   0 Batch  940/3125   train_loss = 1.439\n",
      "2017-12-16T13:42:06.801228: Epoch   0 Batch  960/3125   train_loss = 1.359\n",
      "2017-12-16T13:42:08.682244: Epoch   0 Batch  980/3125   train_loss = 1.359\n",
      "2017-12-16T13:42:10.542976: Epoch   0 Batch 1000/3125   train_loss = 1.280\n",
      "2017-12-16T13:42:12.428349: Epoch   0 Batch 1020/3125   train_loss = 1.335\n",
      "2017-12-16T13:42:14.304146: Epoch   0 Batch 1040/3125   train_loss = 1.300\n",
      "2017-12-16T13:42:16.129855: Epoch   0 Batch 1060/3125   train_loss = 1.481\n",
      "2017-12-16T13:42:18.042707: Epoch   0 Batch 1080/3125   train_loss = 1.211\n",
      "2017-12-16T13:42:19.894096: Epoch   0 Batch 1100/3125   train_loss = 1.348\n",
      "2017-12-16T13:42:21.786237: Epoch   0 Batch 1120/3125   train_loss = 1.317\n",
      "2017-12-16T13:42:23.779869: Epoch   0 Batch 1140/3125   train_loss = 1.382\n",
      "2017-12-16T13:42:25.658617: Epoch   0 Batch 1160/3125   train_loss = 1.285\n",
      "2017-12-16T13:42:27.537005: Epoch   0 Batch 1180/3125   train_loss = 1.268\n",
      "2017-12-16T13:42:29.415839: Epoch   0 Batch 1200/3125   train_loss = 1.237\n",
      "2017-12-16T13:42:31.264104: Epoch   0 Batch 1220/3125   train_loss = 1.244\n",
      "2017-12-16T13:42:33.102854: Epoch   0 Batch 1240/3125   train_loss = 1.167\n",
      "2017-12-16T13:42:35.010545: Epoch   0 Batch 1260/3125   train_loss = 1.262\n",
      "2017-12-16T13:42:36.924181: Epoch   0 Batch 1280/3125   train_loss = 1.237\n",
      "2017-12-16T13:42:38.860858: Epoch   0 Batch 1300/3125   train_loss = 1.286\n",
      "2017-12-16T13:42:40.789946: Epoch   0 Batch 1320/3125   train_loss = 1.184\n",
      "2017-12-16T13:42:42.690119: Epoch   0 Batch 1340/3125   train_loss = 1.117\n",
      "2017-12-16T13:42:44.549084: Epoch   0 Batch 1360/3125   train_loss = 1.179\n",
      "2017-12-16T13:42:46.417628: Epoch   0 Batch 1380/3125   train_loss = 1.114\n",
      "2017-12-16T13:42:48.342763: Epoch   0 Batch 1400/3125   train_loss = 1.291\n",
      "2017-12-16T13:42:50.142222: Epoch   0 Batch 1420/3125   train_loss = 1.366\n",
      "2017-12-16T13:42:51.978298: Epoch   0 Batch 1440/3125   train_loss = 1.167\n",
      "2017-12-16T13:42:53.808551: Epoch   0 Batch 1460/3125   train_loss = 1.261\n",
      "2017-12-16T13:42:55.700166: Epoch   0 Batch 1480/3125   train_loss = 1.329\n",
      "2017-12-16T13:42:57.539756: Epoch   0 Batch 1500/3125   train_loss = 1.347\n",
      "2017-12-16T13:42:59.446239: Epoch   0 Batch 1520/3125   train_loss = 1.265\n",
      "2017-12-16T13:43:01.328890: Epoch   0 Batch 1540/3125   train_loss = 1.338\n",
      "2017-12-16T13:43:03.219671: Epoch   0 Batch 1560/3125   train_loss = 1.252\n",
      "2017-12-16T13:43:05.064484: Epoch   0 Batch 1580/3125   train_loss = 1.247\n",
      "2017-12-16T13:43:06.928674: Epoch   0 Batch 1600/3125   train_loss = 1.339\n",
      "2017-12-16T13:43:08.803411: Epoch   0 Batch 1620/3125   train_loss = 1.318\n",
      "2017-12-16T13:43:10.704544: Epoch   0 Batch 1640/3125   train_loss = 1.342\n",
      "2017-12-16T13:43:12.503688: Epoch   0 Batch 1660/3125   train_loss = 1.343\n",
      "2017-12-16T13:43:14.373258: Epoch   0 Batch 1680/3125   train_loss = 1.263\n",
      "2017-12-16T13:43:16.265025: Epoch   0 Batch 1700/3125   train_loss = 1.151\n",
      "2017-12-16T13:43:18.134865: Epoch   0 Batch 1720/3125   train_loss = 1.213\n",
      "2017-12-16T13:43:20.013180: Epoch   0 Batch 1740/3125   train_loss = 1.222\n",
      "2017-12-16T13:43:21.899767: Epoch   0 Batch 1760/3125   train_loss = 1.390\n",
      "2017-12-16T13:43:23.802617: Epoch   0 Batch 1780/3125   train_loss = 1.093\n",
      "2017-12-16T13:43:25.691169: Epoch   0 Batch 1800/3125   train_loss = 1.235\n",
      "2017-12-16T13:43:27.523589: Epoch   0 Batch 1820/3125   train_loss = 1.189\n",
      "2017-12-16T13:43:29.383411: Epoch   0 Batch 1840/3125   train_loss = 1.326\n",
      "2017-12-16T13:43:31.258927: Epoch   0 Batch 1860/3125   train_loss = 1.328\n",
      "2017-12-16T13:43:33.052374: Epoch   0 Batch 1880/3125   train_loss = 1.271\n",
      "2017-12-16T13:43:34.955556: Epoch   0 Batch 1900/3125   train_loss = 1.110\n",
      "2017-12-16T13:43:36.822119: Epoch   0 Batch 1920/3125   train_loss = 1.295\n",
      "2017-12-16T13:43:38.821743: Epoch   0 Batch 1940/3125   train_loss = 1.167\n",
      "2017-12-16T13:43:40.693795: Epoch   0 Batch 1960/3125   train_loss = 1.163\n",
      "2017-12-16T13:43:42.568182: Epoch   0 Batch 1980/3125   train_loss = 1.216\n",
      "2017-12-16T13:43:44.438958: Epoch   0 Batch 2000/3125   train_loss = 1.428\n",
      "2017-12-16T13:43:46.274090: Epoch   0 Batch 2020/3125   train_loss = 1.352\n",
      "2017-12-16T13:43:48.088579: Epoch   0 Batch 2040/3125   train_loss = 1.185\n",
      "2017-12-16T13:43:49.982680: Epoch   0 Batch 2060/3125   train_loss = 1.049\n",
      "2017-12-16T13:43:51.890941: Epoch   0 Batch 2080/3125   train_loss = 1.362\n",
      "2017-12-16T13:43:53.712907: Epoch   0 Batch 2100/3125   train_loss = 1.171\n",
      "2017-12-16T13:43:55.614456: Epoch   0 Batch 2120/3125   train_loss = 1.173\n",
      "2017-12-16T13:43:57.510717: Epoch   0 Batch 2140/3125   train_loss = 1.225\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2017-12-16T13:43:59.405302: Epoch   0 Batch 2160/3125   train_loss = 1.161\n",
      "2017-12-16T13:44:01.273483: Epoch   0 Batch 2180/3125   train_loss = 1.200\n",
      "2017-12-16T13:44:03.093218: Epoch   0 Batch 2200/3125   train_loss = 1.224\n",
      "2017-12-16T13:44:04.962788: Epoch   0 Batch 2220/3125   train_loss = 1.196\n",
      "2017-12-16T13:44:06.849350: Epoch   0 Batch 2240/3125   train_loss = 1.025\n",
      "2017-12-16T13:44:08.720523: Epoch   0 Batch 2260/3125   train_loss = 1.232\n",
      "2017-12-16T13:44:10.642303: Epoch   0 Batch 2280/3125   train_loss = 1.248\n",
      "2017-12-16T13:44:12.528397: Epoch   0 Batch 2300/3125   train_loss = 1.206\n",
      "2017-12-16T13:44:14.342143: Epoch   0 Batch 2320/3125   train_loss = 1.434\n",
      "2017-12-16T13:44:16.171160: Epoch   0 Batch 2340/3125   train_loss = 1.318\n",
      "2017-12-16T13:44:18.075941: Epoch   0 Batch 2360/3125   train_loss = 1.243\n",
      "2017-12-16T13:44:19.958801: Epoch   0 Batch 2380/3125   train_loss = 1.262\n",
      "2017-12-16T13:44:21.807784: Epoch   0 Batch 2400/3125   train_loss = 1.333\n",
      "2017-12-16T13:44:23.707176: Epoch   0 Batch 2420/3125   train_loss = 1.187\n",
      "2017-12-16T13:44:25.568462: Epoch   0 Batch 2440/3125   train_loss = 1.331\n",
      "2017-12-16T13:44:27.443981: Epoch   0 Batch 2460/3125   train_loss = 1.185\n",
      "2017-12-16T13:44:29.325226: Epoch   0 Batch 2480/3125   train_loss = 1.302\n",
      "2017-12-16T13:44:31.149315: Epoch   0 Batch 2500/3125   train_loss = 1.255\n",
      "2017-12-16T13:44:33.031449: Epoch   0 Batch 2520/3125   train_loss = 1.164\n",
      "2017-12-16T13:44:34.867683: Epoch   0 Batch 2540/3125   train_loss = 1.105\n",
      "2017-12-16T13:44:36.745166: Epoch   0 Batch 2560/3125   train_loss = 1.045\n",
      "2017-12-16T13:44:38.730651: Epoch   0 Batch 2580/3125   train_loss = 1.186\n",
      "2017-12-16T13:44:41.910534: Epoch   0 Batch 2600/3125   train_loss = 1.198\n",
      "2017-12-16T13:44:43.996344: Epoch   0 Batch 2620/3125   train_loss = 1.114\n",
      "2017-12-16T13:44:45.961293: Epoch   0 Batch 2640/3125   train_loss = 1.189\n",
      "2017-12-16T13:44:47.819447: Epoch   0 Batch 2660/3125   train_loss = 1.306\n",
      "2017-12-16T13:44:49.724466: Epoch   0 Batch 2680/3125   train_loss = 1.128\n",
      "2017-12-16T13:44:51.590890: Epoch   0 Batch 2700/3125   train_loss = 1.309\n",
      "2017-12-16T13:44:53.441962: Epoch   0 Batch 2720/3125   train_loss = 1.185\n",
      "2017-12-16T13:44:55.357901: Epoch   0 Batch 2740/3125   train_loss = 1.219\n",
      "2017-12-16T13:44:57.352801: Epoch   0 Batch 2760/3125   train_loss = 1.225\n",
      "2017-12-16T13:44:59.150060: Epoch   0 Batch 2780/3125   train_loss = 1.196\n",
      "2017-12-16T13:45:01.030421: Epoch   0 Batch 2800/3125   train_loss = 1.403\n",
      "2017-12-16T13:45:02.900104: Epoch   0 Batch 2820/3125   train_loss = 1.453\n",
      "2017-12-16T13:45:04.767343: Epoch   0 Batch 2840/3125   train_loss = 1.214\n",
      "2017-12-16T13:45:06.632400: Epoch   0 Batch 2860/3125   train_loss = 1.200\n",
      "2017-12-16T13:45:08.516712: Epoch   0 Batch 2880/3125   train_loss = 1.270\n",
      "2017-12-16T13:45:10.398296: Epoch   0 Batch 2900/3125   train_loss = 1.253\n",
      "2017-12-16T13:45:12.280877: Epoch   0 Batch 2920/3125   train_loss = 1.192\n",
      "2017-12-16T13:45:14.101760: Epoch   0 Batch 2940/3125   train_loss = 1.216\n",
      "2017-12-16T13:45:15.960747: Epoch   0 Batch 2960/3125   train_loss = 1.234\n",
      "2017-12-16T13:45:17.878291: Epoch   0 Batch 2980/3125   train_loss = 1.249\n",
      "2017-12-16T13:45:19.926368: Epoch   0 Batch 3000/3125   train_loss = 1.228\n",
      "2017-12-16T13:45:22.167643: Epoch   0 Batch 3020/3125   train_loss = 1.271\n",
      "2017-12-16T13:45:24.509070: Epoch   0 Batch 3040/3125   train_loss = 1.275\n",
      "2017-12-16T13:45:26.515409: Epoch   0 Batch 3060/3125   train_loss = 1.258\n",
      "2017-12-16T13:45:28.449037: Epoch   0 Batch 3080/3125   train_loss = 1.322\n",
      "2017-12-16T13:45:30.597746: Epoch   0 Batch 3100/3125   train_loss = 1.253\n",
      "2017-12-16T13:45:32.521245: Epoch   0 Batch 3120/3125   train_loss = 1.127\n",
      "2017-12-16T13:45:32.943383: Epoch   0 Batch    0/781   test_loss = 1.006\n",
      "2017-12-16T13:45:33.289002: Epoch   0 Batch   20/781   test_loss = 1.170\n",
      "2017-12-16T13:45:33.923827: Epoch   0 Batch   40/781   test_loss = 1.213\n",
      "2017-12-16T13:45:34.364524: Epoch   0 Batch   60/781   test_loss = 1.333\n",
      "2017-12-16T13:45:34.712000: Epoch   0 Batch   80/781   test_loss = 1.391\n",
      "2017-12-16T13:45:35.000222: Epoch   0 Batch  100/781   test_loss = 1.398\n",
      "2017-12-16T13:45:35.486534: Epoch   0 Batch  120/781   test_loss = 1.290\n",
      "2017-12-16T13:45:35.847098: Epoch   0 Batch  140/781   test_loss = 1.273\n",
      "2017-12-16T13:45:36.136884: Epoch   0 Batch  160/781   test_loss = 1.478\n",
      "2017-12-16T13:45:36.495139: Epoch   0 Batch  180/781   test_loss = 1.327\n",
      "2017-12-16T13:45:36.807935: Epoch   0 Batch  200/781   test_loss = 1.311\n",
      "2017-12-16T13:45:37.377732: Epoch   0 Batch  220/781   test_loss = 1.076\n",
      "2017-12-16T13:45:37.730602: Epoch   0 Batch  240/781   test_loss = 1.234\n",
      "2017-12-16T13:45:38.024326: Epoch   0 Batch  260/781   test_loss = 1.277\n",
      "2017-12-16T13:45:38.521149: Epoch   0 Batch  280/781   test_loss = 1.513\n",
      "2017-12-16T13:45:38.926469: Epoch   0 Batch  300/781   test_loss = 1.245\n",
      "2017-12-16T13:45:39.286746: Epoch   0 Batch  320/781   test_loss = 1.400\n",
      "2017-12-16T13:45:39.584400: Epoch   0 Batch  340/781   test_loss = 0.976\n",
      "2017-12-16T13:45:39.911867: Epoch   0 Batch  360/781   test_loss = 1.372\n",
      "2017-12-16T13:45:40.250978: Epoch   0 Batch  380/781   test_loss = 1.230\n",
      "2017-12-16T13:45:40.542244: Epoch   0 Batch  400/781   test_loss = 1.245\n",
      "2017-12-16T13:45:40.834520: Epoch   0 Batch  420/781   test_loss = 1.109\n",
      "2017-12-16T13:45:41.126793: Epoch   0 Batch  440/781   test_loss = 1.292\n",
      "2017-12-16T13:45:41.459652: Epoch   0 Batch  460/781   test_loss = 1.189\n",
      "2017-12-16T13:45:41.758404: Epoch   0 Batch  480/781   test_loss = 1.219\n",
      "2017-12-16T13:45:42.045974: Epoch   0 Batch  500/781   test_loss = 1.100\n",
      "2017-12-16T13:45:42.386796: Epoch   0 Batch  520/781   test_loss = 1.278\n",
      "2017-12-16T13:45:42.685549: Epoch   0 Batch  540/781   test_loss = 1.136\n",
      "2017-12-16T13:45:42.976353: Epoch   0 Batch  560/781   test_loss = 1.477\n",
      "2017-12-16T13:45:43.305586: Epoch   0 Batch  580/781   test_loss = 1.263\n",
      "2017-12-16T13:45:43.603854: Epoch   0 Batch  600/781   test_loss = 1.271\n",
      "2017-12-16T13:45:43.902248: Epoch   0 Batch  620/781   test_loss = 1.386\n",
      "2017-12-16T13:45:44.234453: Epoch   0 Batch  640/781   test_loss = 1.325\n",
      "2017-12-16T13:45:44.528212: Epoch   0 Batch  660/781   test_loss = 1.272\n",
      "2017-12-16T13:45:44.820007: Epoch   0 Batch  680/781   test_loss = 1.581\n",
      "2017-12-16T13:45:45.106127: Epoch   0 Batch  700/781   test_loss = 1.195\n",
      "2017-12-16T13:45:45.445179: Epoch   0 Batch  720/781   test_loss = 1.368\n",
      "2017-12-16T13:45:45.739010: Epoch   0 Batch  740/781   test_loss = 1.268\n",
      "2017-12-16T13:45:46.025527: Epoch   0 Batch  760/781   test_loss = 1.229\n",
      "2017-12-16T13:45:46.360655: Epoch   0 Batch  780/781   test_loss = 1.264\n",
      "2017-12-16T13:45:48.488109: Epoch   1 Batch   15/3125   train_loss = 1.277\n",
      "2017-12-16T13:45:50.439863: Epoch   1 Batch   35/3125   train_loss = 1.212\n",
      "2017-12-16T13:45:52.623315: Epoch   1 Batch   55/3125   train_loss = 1.307\n",
      "2017-12-16T13:45:55.366593: Epoch   1 Batch   75/3125   train_loss = 1.194\n",
      "2017-12-16T13:45:58.409762: Epoch   1 Batch   95/3125   train_loss = 1.156\n",
      "2017-12-16T13:46:00.904934: Epoch   1 Batch  115/3125   train_loss = 1.199\n",
      "2017-12-16T13:46:03.448548: Epoch   1 Batch  135/3125   train_loss = 1.092\n",
      "2017-12-16T13:46:06.089356: Epoch   1 Batch  155/3125   train_loss = 1.136\n",
      "2017-12-16T13:46:08.975568: Epoch   1 Batch  175/3125   train_loss = 1.198\n",
      "2017-12-16T13:46:11.433956: Epoch   1 Batch  195/3125   train_loss = 1.253\n",
      "2017-12-16T13:46:13.618465: Epoch   1 Batch  215/3125   train_loss = 1.148\n",
      "2017-12-16T13:46:16.428351: Epoch   1 Batch  235/3125   train_loss = 1.122\n",
      "2017-12-16T13:46:19.625856: Epoch   1 Batch  255/3125   train_loss = 1.274\n",
      "2017-12-16T13:46:22.222086: Epoch   1 Batch  275/3125   train_loss = 1.117\n",
      "2017-12-16T13:46:24.630762: Epoch   1 Batch  295/3125   train_loss = 1.066\n",
      "2017-12-16T13:46:26.832794: Epoch   1 Batch  315/3125   train_loss = 1.148\n",
      "2017-12-16T13:46:29.976923: Epoch   1 Batch  335/3125   train_loss = 1.081\n",
      "2017-12-16T13:46:33.357041: Epoch   1 Batch  355/3125   train_loss = 1.279\n",
      "2017-12-16T13:46:35.466430: Epoch   1 Batch  375/3125   train_loss = 1.244\n",
      "2017-12-16T13:46:37.365799: Epoch   1 Batch  395/3125   train_loss = 1.112\n",
      "2017-12-16T13:46:39.278989: Epoch   1 Batch  415/3125   train_loss = 1.392\n",
      "2017-12-16T13:46:43.565288: Epoch   1 Batch  435/3125   train_loss = 1.243\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2017-12-16T13:46:47.418658: Epoch   1 Batch  455/3125   train_loss = 1.173\n",
      "2017-12-16T13:46:49.467236: Epoch   1 Batch  475/3125   train_loss = 1.279\n",
      "2017-12-16T13:46:52.799696: Epoch   1 Batch  495/3125   train_loss = 1.171\n",
      "2017-12-16T13:46:54.686050: Epoch   1 Batch  515/3125   train_loss = 1.228\n",
      "2017-12-16T13:46:56.586085: Epoch   1 Batch  535/3125   train_loss = 1.256\n",
      "2017-12-16T13:46:58.432347: Epoch   1 Batch  555/3125   train_loss = 1.311\n",
      "2017-12-16T13:47:01.269217: Epoch   1 Batch  575/3125   train_loss = 1.209\n",
      "2017-12-16T13:47:03.609999: Epoch   1 Batch  595/3125   train_loss = 1.324\n",
      "2017-12-16T13:47:05.452957: Epoch   1 Batch  615/3125   train_loss = 1.120\n",
      "2017-12-16T13:47:07.276559: Epoch   1 Batch  635/3125   train_loss = 1.172\n",
      "2017-12-16T13:47:09.083714: Epoch   1 Batch  655/3125   train_loss = 1.118\n",
      "2017-12-16T13:47:10.963211: Epoch   1 Batch  675/3125   train_loss = 0.977\n",
      "2017-12-16T13:47:12.975266: Epoch   1 Batch  695/3125   train_loss = 1.183\n",
      "2017-12-16T13:47:14.861310: Epoch   1 Batch  715/3125   train_loss = 1.251\n",
      "2017-12-16T13:47:16.781217: Epoch   1 Batch  735/3125   train_loss = 1.042\n",
      "2017-12-16T13:47:18.691300: Epoch   1 Batch  755/3125   train_loss = 1.236\n",
      "2017-12-16T13:47:20.786723: Epoch   1 Batch  775/3125   train_loss = 1.060\n",
      "2017-12-16T13:47:22.707628: Epoch   1 Batch  795/3125   train_loss = 1.396\n",
      "2017-12-16T13:47:24.989078: Epoch   1 Batch  815/3125   train_loss = 1.227\n",
      "2017-12-16T13:47:27.093072: Epoch   1 Batch  835/3125   train_loss = 1.196\n",
      "2017-12-16T13:47:29.444656: Epoch   1 Batch  855/3125   train_loss = 1.341\n",
      "2017-12-16T13:47:31.719755: Epoch   1 Batch  875/3125   train_loss = 1.297\n",
      "2017-12-16T13:47:34.686454: Epoch   1 Batch  895/3125   train_loss = 1.221\n",
      "2017-12-16T13:47:37.348780: Epoch   1 Batch  915/3125   train_loss = 1.193\n",
      "2017-12-16T13:47:40.400641: Epoch   1 Batch  935/3125   train_loss = 1.307\n",
      "2017-12-16T13:47:43.646934: Epoch   1 Batch  955/3125   train_loss = 1.284\n",
      "2017-12-16T13:47:45.590257: Epoch   1 Batch  975/3125   train_loss = 1.258\n",
      "2017-12-16T13:47:47.746050: Epoch   1 Batch  995/3125   train_loss = 0.965\n",
      "2017-12-16T13:47:49.632294: Epoch   1 Batch 1015/3125   train_loss = 1.201\n",
      "2017-12-16T13:47:51.511914: Epoch   1 Batch 1035/3125   train_loss = 1.138\n",
      "2017-12-16T13:47:53.412235: Epoch   1 Batch 1055/3125   train_loss = 1.231\n",
      "2017-12-16T13:47:55.259671: Epoch   1 Batch 1075/3125   train_loss = 1.089\n",
      "2017-12-16T13:47:57.058144: Epoch   1 Batch 1095/3125   train_loss = 1.210\n",
      "2017-12-16T13:47:58.892800: Epoch   1 Batch 1115/3125   train_loss = 1.196\n",
      "2017-12-16T13:48:00.714795: Epoch   1 Batch 1135/3125   train_loss = 1.193\n",
      "2017-12-16T13:48:02.543479: Epoch   1 Batch 1155/3125   train_loss = 1.223\n",
      "2017-12-16T13:48:04.443435: Epoch   1 Batch 1175/3125   train_loss = 1.266\n",
      "2017-12-16T13:48:06.348656: Epoch   1 Batch 1195/3125   train_loss = 1.404\n",
      "2017-12-16T13:48:08.179017: Epoch   1 Batch 1215/3125   train_loss = 1.150\n",
      "2017-12-16T13:48:10.154654: Epoch   1 Batch 1235/3125   train_loss = 1.199\n",
      "2017-12-16T13:48:11.993505: Epoch   1 Batch 1255/3125   train_loss = 1.107\n",
      "2017-12-16T13:48:13.879828: Epoch   1 Batch 1275/3125   train_loss = 1.103\n",
      "2017-12-16T13:48:15.714811: Epoch   1 Batch 1295/3125   train_loss = 1.228\n",
      "2017-12-16T13:48:17.557117: Epoch   1 Batch 1315/3125   train_loss = 1.279\n",
      "2017-12-16T13:48:19.441510: Epoch   1 Batch 1335/3125   train_loss = 1.120\n",
      "2017-12-16T13:48:21.287049: Epoch   1 Batch 1355/3125   train_loss = 1.155\n",
      "2017-12-16T13:48:23.129560: Epoch   1 Batch 1375/3125   train_loss = 1.264\n",
      "2017-12-16T13:48:24.993998: Epoch   1 Batch 1395/3125   train_loss = 1.225\n",
      "2017-12-16T13:48:26.886874: Epoch   1 Batch 1415/3125   train_loss = 1.170\n",
      "2017-12-16T13:48:28.776566: Epoch   1 Batch 1435/3125   train_loss = 1.315\n",
      "2017-12-16T13:48:30.639073: Epoch   1 Batch 1455/3125   train_loss = 1.299\n",
      "2017-12-16T13:48:32.553285: Epoch   1 Batch 1475/3125   train_loss = 1.219\n",
      "2017-12-16T13:48:34.435214: Epoch   1 Batch 1495/3125   train_loss = 1.102\n",
      "2017-12-16T13:48:36.287466: Epoch   1 Batch 1515/3125   train_loss = 1.148\n",
      "2017-12-16T13:48:38.142857: Epoch   1 Batch 1535/3125   train_loss = 0.978\n",
      "2017-12-16T13:48:40.066985: Epoch   1 Batch 1555/3125   train_loss = 1.287\n",
      "2017-12-16T13:48:41.903133: Epoch   1 Batch 1575/3125   train_loss = 1.222\n",
      "2017-12-16T13:48:43.803382: Epoch   1 Batch 1595/3125   train_loss = 1.220\n",
      "2017-12-16T13:48:45.666384: Epoch   1 Batch 1615/3125   train_loss = 1.151\n",
      "2017-12-16T13:48:47.521072: Epoch   1 Batch 1635/3125   train_loss = 1.172\n",
      "2017-12-16T13:48:49.430547: Epoch   1 Batch 1655/3125   train_loss = 1.271\n",
      "2017-12-16T13:48:51.298206: Epoch   1 Batch 1675/3125   train_loss = 1.104\n",
      "2017-12-16T13:48:53.115083: Epoch   1 Batch 1695/3125   train_loss = 1.195\n",
      "2017-12-16T13:48:54.990755: Epoch   1 Batch 1715/3125   train_loss = 1.176\n",
      "2017-12-16T13:48:56.859542: Epoch   1 Batch 1735/3125   train_loss = 1.363\n",
      "2017-12-16T13:48:58.737334: Epoch   1 Batch 1755/3125   train_loss = 1.192\n",
      "2017-12-16T13:49:00.613864: Epoch   1 Batch 1775/3125   train_loss = 1.149\n",
      "2017-12-16T13:49:02.497381: Epoch   1 Batch 1795/3125   train_loss = 1.199\n",
      "2017-12-16T13:49:04.369548: Epoch   1 Batch 1815/3125   train_loss = 1.108\n",
      "2017-12-16T13:49:06.264604: Epoch   1 Batch 1835/3125   train_loss = 1.306\n",
      "2017-12-16T13:49:08.063295: Epoch   1 Batch 1855/3125   train_loss = 1.184\n",
      "2017-12-16T13:49:09.882167: Epoch   1 Batch 1875/3125   train_loss = 1.341\n",
      "2017-12-16T13:49:11.705181: Epoch   1 Batch 1895/3125   train_loss = 1.151\n",
      "2017-12-16T13:49:13.579193: Epoch   1 Batch 1915/3125   train_loss = 1.079\n",
      "2017-12-16T13:49:15.497861: Epoch   1 Batch 1935/3125   train_loss = 1.190\n",
      "2017-12-16T13:49:17.373922: Epoch   1 Batch 1955/3125   train_loss = 1.084\n",
      "2017-12-16T13:49:19.254930: Epoch   1 Batch 1975/3125   train_loss = 1.113\n",
      "2017-12-16T13:49:21.063182: Epoch   1 Batch 1995/3125   train_loss = 1.284\n",
      "2017-12-16T13:49:22.971524: Epoch   1 Batch 2015/3125   train_loss = 1.325\n",
      "2017-12-16T13:49:24.941264: Epoch   1 Batch 2035/3125   train_loss = 1.194\n",
      "2017-12-16T13:49:26.806955: Epoch   1 Batch 2055/3125   train_loss = 1.160\n",
      "2017-12-16T13:49:28.627787: Epoch   1 Batch 2075/3125   train_loss = 1.233\n",
      "2017-12-16T13:49:30.438611: Epoch   1 Batch 2095/3125   train_loss = 1.088\n",
      "2017-12-16T13:49:32.275009: Epoch   1 Batch 2115/3125   train_loss = 1.253\n",
      "2017-12-16T13:49:34.123965: Epoch   1 Batch 2135/3125   train_loss = 1.084\n",
      "2017-12-16T13:49:36.001022: Epoch   1 Batch 2155/3125   train_loss = 1.176\n",
      "2017-12-16T13:49:37.890046: Epoch   1 Batch 2175/3125   train_loss = 1.178\n",
      "2017-12-16T13:49:39.732387: Epoch   1 Batch 2195/3125   train_loss = 1.246\n",
      "2017-12-16T13:49:41.622695: Epoch   1 Batch 2215/3125   train_loss = 1.238\n",
      "2017-12-16T13:49:43.495758: Epoch   1 Batch 2235/3125   train_loss = 1.341\n",
      "2017-12-16T13:49:45.361881: Epoch   1 Batch 2255/3125   train_loss = 1.200\n",
      "2017-12-16T13:49:47.225657: Epoch   1 Batch 2275/3125   train_loss = 1.129\n",
      "2017-12-16T13:49:49.055511: Epoch   1 Batch 2295/3125   train_loss = 1.460\n",
      "2017-12-16T13:49:50.906412: Epoch   1 Batch 2315/3125   train_loss = 1.371\n",
      "2017-12-16T13:49:54.108193: Epoch   1 Batch 2335/3125   train_loss = 1.183\n",
      "2017-12-16T13:49:57.186127: Epoch   1 Batch 2355/3125   train_loss = 1.155\n",
      "2017-12-16T13:49:59.630553: Epoch   1 Batch 2375/3125   train_loss = 1.417\n",
      "2017-12-16T13:50:01.518848: Epoch   1 Batch 2395/3125   train_loss = 1.232\n",
      "2017-12-16T13:50:03.410090: Epoch   1 Batch 2415/3125   train_loss = 1.221\n",
      "2017-12-16T13:50:05.280971: Epoch   1 Batch 2435/3125   train_loss = 1.097\n",
      "2017-12-16T13:50:07.122339: Epoch   1 Batch 2455/3125   train_loss = 1.246\n",
      "2017-12-16T13:50:08.964534: Epoch   1 Batch 2475/3125   train_loss = 1.011\n",
      "2017-12-16T13:50:11.546618: Epoch   1 Batch 2495/3125   train_loss = 1.088\n",
      "2017-12-16T13:50:13.428724: Epoch   1 Batch 2515/3125   train_loss = 1.205\n",
      "2017-12-16T13:50:15.282496: Epoch   1 Batch 2535/3125   train_loss = 1.302\n",
      "2017-12-16T13:50:17.084289: Epoch   1 Batch 2555/3125   train_loss = 1.077\n",
      "2017-12-16T13:50:18.996458: Epoch   1 Batch 2575/3125   train_loss = 1.044\n",
      "2017-12-16T13:50:20.907255: Epoch   1 Batch 2595/3125   train_loss = 1.209\n",
      "2017-12-16T13:50:22.904527: Epoch   1 Batch 2615/3125   train_loss = 1.231\n",
      "2017-12-16T13:50:24.823338: Epoch   1 Batch 2635/3125   train_loss = 1.093\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2017-12-16T13:50:26.909092: Epoch   1 Batch 2655/3125   train_loss = 1.193\n",
      "2017-12-16T13:50:28.836917: Epoch   1 Batch 2675/3125   train_loss = 1.128\n",
      "2017-12-16T13:50:30.709264: Epoch   1 Batch 2695/3125   train_loss = 1.266\n",
      "2017-12-16T13:50:32.591310: Epoch   1 Batch 2715/3125   train_loss = 1.205\n",
      "2017-12-16T13:50:34.573507: Epoch   1 Batch 2735/3125   train_loss = 0.993\n",
      "2017-12-16T13:50:36.450380: Epoch   1 Batch 2755/3125   train_loss = 1.129\n",
      "2017-12-16T13:50:38.349065: Epoch   1 Batch 2775/3125   train_loss = 1.250\n",
      "2017-12-16T13:50:40.280534: Epoch   1 Batch 2795/3125   train_loss = 1.190\n",
      "2017-12-16T13:50:42.138165: Epoch   1 Batch 2815/3125   train_loss = 1.185\n",
      "2017-12-16T13:50:44.095689: Epoch   1 Batch 2835/3125   train_loss = 1.163\n",
      "2017-12-16T13:50:45.978958: Epoch   1 Batch 2855/3125   train_loss = 1.218\n",
      "2017-12-16T13:50:47.863159: Epoch   1 Batch 2875/3125   train_loss = 1.292\n",
      "2017-12-16T13:50:49.742412: Epoch   1 Batch 2895/3125   train_loss = 1.224\n",
      "2017-12-16T13:50:51.606064: Epoch   1 Batch 2915/3125   train_loss = 1.070\n",
      "2017-12-16T13:50:53.505053: Epoch   1 Batch 2935/3125   train_loss = 1.211\n",
      "2017-12-16T13:50:55.389122: Epoch   1 Batch 2955/3125   train_loss = 1.289\n",
      "2017-12-16T13:50:57.257098: Epoch   1 Batch 2975/3125   train_loss = 1.192\n",
      "2017-12-16T13:51:00.079149: Epoch   1 Batch 2995/3125   train_loss = 1.155\n",
      "2017-12-16T13:51:02.109303: Epoch   1 Batch 3015/3125   train_loss = 1.194\n",
      "2017-12-16T13:51:04.046348: Epoch   1 Batch 3035/3125   train_loss = 1.280\n",
      "2017-12-16T13:51:06.083184: Epoch   1 Batch 3055/3125   train_loss = 1.243\n",
      "2017-12-16T13:51:08.057927: Epoch   1 Batch 3075/3125   train_loss = 1.142\n",
      "2017-12-16T13:51:10.091180: Epoch   1 Batch 3095/3125   train_loss = 1.169\n",
      "2017-12-16T13:51:12.564887: Epoch   1 Batch 3115/3125   train_loss = 1.012\n",
      "2017-12-16T13:51:14.260537: Epoch   1 Batch   19/781   test_loss = 1.220\n",
      "2017-12-16T13:51:14.608840: Epoch   1 Batch   39/781   test_loss = 1.082\n",
      "2017-12-16T13:51:15.029500: Epoch   1 Batch   59/781   test_loss = 1.064\n",
      "2017-12-16T13:51:15.486881: Epoch   1 Batch   79/781   test_loss = 1.145\n",
      "2017-12-16T13:51:15.870695: Epoch   1 Batch   99/781   test_loss = 1.117\n",
      "2017-12-16T13:51:16.300955: Epoch   1 Batch  119/781   test_loss = 1.225\n",
      "2017-12-16T13:51:16.586779: Epoch   1 Batch  139/781   test_loss = 1.239\n",
      "2017-12-16T13:51:16.939068: Epoch   1 Batch  159/781   test_loss = 1.164\n",
      "2017-12-16T13:51:17.249986: Epoch   1 Batch  179/781   test_loss = 1.243\n",
      "2017-12-16T13:51:17.691472: Epoch   1 Batch  199/781   test_loss = 1.134\n",
      "2017-12-16T13:51:18.000131: Epoch   1 Batch  219/781   test_loss = 1.178\n",
      "2017-12-16T13:51:18.512451: Epoch   1 Batch  239/781   test_loss = 1.241\n",
      "2017-12-16T13:51:18.983509: Epoch   1 Batch  259/781   test_loss = 1.147\n",
      "2017-12-16T13:51:19.379491: Epoch   1 Batch  279/781   test_loss = 1.357\n",
      "2017-12-16T13:51:19.747703: Epoch   1 Batch  299/781   test_loss = 1.419\n",
      "2017-12-16T13:51:20.244891: Epoch   1 Batch  319/781   test_loss = 1.218\n",
      "2017-12-16T13:51:20.624569: Epoch   1 Batch  339/781   test_loss = 1.070\n",
      "2017-12-16T13:51:20.919223: Epoch   1 Batch  359/781   test_loss = 1.013\n",
      "2017-12-16T13:51:21.467644: Epoch   1 Batch  379/781   test_loss = 1.254\n",
      "2017-12-16T13:51:21.960366: Epoch   1 Batch  399/781   test_loss = 1.103\n",
      "2017-12-16T13:51:22.431525: Epoch   1 Batch  419/781   test_loss = 1.151\n",
      "2017-12-16T13:51:22.966418: Epoch   1 Batch  439/781   test_loss = 1.291\n",
      "2017-12-16T13:51:23.389274: Epoch   1 Batch  459/781   test_loss = 1.244\n",
      "2017-12-16T13:51:23.683180: Epoch   1 Batch  479/781   test_loss = 1.348\n",
      "2017-12-16T13:51:24.021234: Epoch   1 Batch  499/781   test_loss = 1.091\n",
      "2017-12-16T13:51:24.480139: Epoch   1 Batch  519/781   test_loss = 1.318\n",
      "2017-12-16T13:51:24.876309: Epoch   1 Batch  539/781   test_loss = 1.035\n",
      "2017-12-16T13:51:25.415006: Epoch   1 Batch  559/781   test_loss = 1.400\n",
      "2017-12-16T13:51:25.719627: Epoch   1 Batch  579/781   test_loss = 1.294\n",
      "2017-12-16T13:51:26.051437: Epoch   1 Batch  599/781   test_loss = 1.126\n",
      "2017-12-16T13:51:26.529656: Epoch   1 Batch  619/781   test_loss = 1.292\n",
      "2017-12-16T13:51:27.016844: Epoch   1 Batch  639/781   test_loss = 1.141\n",
      "2017-12-16T13:51:27.501833: Epoch   1 Batch  659/781   test_loss = 1.359\n",
      "2017-12-16T13:51:27.793654: Epoch   1 Batch  679/781   test_loss = 1.300\n",
      "2017-12-16T13:51:28.140575: Epoch   1 Batch  699/781   test_loss = 1.055\n",
      "2017-12-16T13:51:28.483967: Epoch   1 Batch  719/781   test_loss = 1.160\n",
      "2017-12-16T13:51:28.770431: Epoch   1 Batch  739/781   test_loss = 1.210\n",
      "2017-12-16T13:51:29.052141: Epoch   1 Batch  759/781   test_loss = 1.132\n",
      "2017-12-16T13:51:29.381523: Epoch   1 Batch  779/781   test_loss = 1.066\n",
      "2017-12-16T13:51:31.625935: Epoch   2 Batch   10/3125   train_loss = 1.049\n",
      "2017-12-16T13:51:33.768094: Epoch   2 Batch   30/3125   train_loss = 1.181\n",
      "2017-12-16T13:51:36.937264: Epoch   2 Batch   50/3125   train_loss = 1.184\n",
      "2017-12-16T13:51:39.906366: Epoch   2 Batch   70/3125   train_loss = 1.230\n",
      "2017-12-16T13:51:42.344202: Epoch   2 Batch   90/3125   train_loss = 1.236\n",
      "2017-12-16T13:51:45.189740: Epoch   2 Batch  110/3125   train_loss = 1.107\n",
      "2017-12-16T13:51:47.648726: Epoch   2 Batch  130/3125   train_loss = 1.086\n",
      "2017-12-16T13:51:49.743800: Epoch   2 Batch  150/3125   train_loss = 1.260\n",
      "2017-12-16T13:51:51.673385: Epoch   2 Batch  170/3125   train_loss = 1.083\n",
      "2017-12-16T13:51:53.809294: Epoch   2 Batch  190/3125   train_loss = 1.276\n",
      "2017-12-16T13:51:55.698467: Epoch   2 Batch  210/3125   train_loss = 1.133\n",
      "2017-12-16T13:51:57.595753: Epoch   2 Batch  230/3125   train_loss = 1.132\n",
      "2017-12-16T13:52:00.325104: Epoch   2 Batch  250/3125   train_loss = 1.096\n",
      "2017-12-16T13:52:03.025900: Epoch   2 Batch  270/3125   train_loss = 1.027\n",
      "2017-12-16T13:52:06.281145: Epoch   2 Batch  290/3125   train_loss = 1.178\n",
      "2017-12-16T13:52:08.261389: Epoch   2 Batch  310/3125   train_loss = 1.192\n",
      "2017-12-16T13:52:10.111954: Epoch   2 Batch  330/3125   train_loss = 1.227\n",
      "2017-12-16T13:52:12.010406: Epoch   2 Batch  350/3125   train_loss = 1.034\n",
      "2017-12-16T13:52:13.966479: Epoch   2 Batch  370/3125   train_loss = 1.279\n",
      "2017-12-16T13:52:15.822754: Epoch   2 Batch  390/3125   train_loss = 1.353\n",
      "2017-12-16T13:52:17.739438: Epoch   2 Batch  410/3125   train_loss = 1.110\n",
      "2017-12-16T13:52:19.632027: Epoch   2 Batch  430/3125   train_loss = 1.325\n",
      "2017-12-16T13:52:21.544628: Epoch   2 Batch  450/3125   train_loss = 1.089\n",
      "2017-12-16T13:52:23.459669: Epoch   2 Batch  470/3125   train_loss = 1.015\n",
      "2017-12-16T13:52:25.353603: Epoch   2 Batch  490/3125   train_loss = 1.218\n",
      "2017-12-16T13:52:27.185907: Epoch   2 Batch  510/3125   train_loss = 1.233\n",
      "2017-12-16T13:52:29.194471: Epoch   2 Batch  530/3125   train_loss = 1.144\n",
      "2017-12-16T13:52:31.066785: Epoch   2 Batch  550/3125   train_loss = 1.102\n",
      "2017-12-16T13:52:32.959964: Epoch   2 Batch  570/3125   train_loss = 1.300\n",
      "2017-12-16T13:52:34.849023: Epoch   2 Batch  590/3125   train_loss = 1.182\n",
      "2017-12-16T13:52:36.751245: Epoch   2 Batch  610/3125   train_loss = 1.130\n",
      "2017-12-16T13:52:38.714822: Epoch   2 Batch  630/3125   train_loss = 1.112\n",
      "2017-12-16T13:52:40.622148: Epoch   2 Batch  650/3125   train_loss = 1.240\n",
      "2017-12-16T13:52:42.565146: Epoch   2 Batch  670/3125   train_loss = 1.073\n",
      "2017-12-16T13:52:44.447104: Epoch   2 Batch  690/3125   train_loss = 1.176\n",
      "2017-12-16T13:52:46.337059: Epoch   2 Batch  710/3125   train_loss = 1.109\n",
      "2017-12-16T13:52:48.230715: Epoch   2 Batch  730/3125   train_loss = 1.052\n",
      "2017-12-16T13:52:50.162023: Epoch   2 Batch  750/3125   train_loss = 1.156\n",
      "2017-12-16T13:52:52.036104: Epoch   2 Batch  770/3125   train_loss = 1.067\n",
      "2017-12-16T13:52:53.925343: Epoch   2 Batch  790/3125   train_loss = 1.153\n",
      "2017-12-16T13:52:55.827658: Epoch   2 Batch  810/3125   train_loss = 0.999\n",
      "2017-12-16T13:52:57.732637: Epoch   2 Batch  830/3125   train_loss = 1.018\n",
      "2017-12-16T13:52:59.605211: Epoch   2 Batch  850/3125   train_loss = 1.221\n",
      "2017-12-16T13:53:01.498394: Epoch   2 Batch  870/3125   train_loss = 1.030\n",
      "2017-12-16T13:53:03.384207: Epoch   2 Batch  890/3125   train_loss = 1.032\n",
      "2017-12-16T13:53:05.283149: Epoch   2 Batch  910/3125   train_loss = 1.214\n",
      "2017-12-16T13:53:07.119885: Epoch   2 Batch  930/3125   train_loss = 1.271\n",
      "2017-12-16T13:53:09.057828: Epoch   2 Batch  950/3125   train_loss = 1.085\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2017-12-16T13:53:10.975506: Epoch   2 Batch  970/3125   train_loss = 1.344\n",
      "2017-12-16T13:53:12.875230: Epoch   2 Batch  990/3125   train_loss = 1.013\n",
      "2017-12-16T13:53:14.785820: Epoch   2 Batch 1010/3125   train_loss = 1.363\n",
      "2017-12-16T13:53:16.681606: Epoch   2 Batch 1030/3125   train_loss = 1.105\n",
      "2017-12-16T13:53:18.598939: Epoch   2 Batch 1050/3125   train_loss = 1.159\n",
      "2017-12-16T13:53:20.517956: Epoch   2 Batch 1070/3125   train_loss = 1.170\n",
      "2017-12-16T13:53:22.429218: Epoch   2 Batch 1090/3125   train_loss = 1.226\n",
      "2017-12-16T13:53:24.283639: Epoch   2 Batch 1110/3125   train_loss = 1.341\n",
      "2017-12-16T13:53:26.154716: Epoch   2 Batch 1130/3125   train_loss = 1.100\n",
      "2017-12-16T13:53:29.227157: Epoch   2 Batch 1150/3125   train_loss = 1.175\n",
      "2017-12-16T13:53:31.362334: Epoch   2 Batch 1170/3125   train_loss = 1.167\n",
      "2017-12-16T13:53:33.971771: Epoch   2 Batch 1190/3125   train_loss = 1.275\n",
      "2017-12-16T13:53:36.431524: Epoch   2 Batch 1210/3125   train_loss = 1.114\n",
      "2017-12-16T13:53:39.397698: Epoch   2 Batch 1230/3125   train_loss = 1.101\n",
      "2017-12-16T13:53:41.976588: Epoch   2 Batch 1250/3125   train_loss = 1.120\n",
      "2017-12-16T13:53:44.475130: Epoch   2 Batch 1270/3125   train_loss = 1.184\n",
      "2017-12-16T13:53:46.598561: Epoch   2 Batch 1290/3125   train_loss = 1.140\n",
      "2017-12-16T13:53:48.490082: Epoch   2 Batch 1310/3125   train_loss = 1.147\n",
      "2017-12-16T13:53:50.838767: Epoch   2 Batch 1330/3125   train_loss = 1.335\n",
      "2017-12-16T13:53:53.952693: Epoch   2 Batch 1350/3125   train_loss = 1.068\n",
      "2017-12-16T13:53:56.770754: Epoch   2 Batch 1370/3125   train_loss = 1.032\n",
      "2017-12-16T13:53:58.718471: Epoch   2 Batch 1390/3125   train_loss = 1.166\n",
      "2017-12-16T13:54:01.145101: Epoch   2 Batch 1410/3125   train_loss = 1.083\n",
      "2017-12-16T13:54:03.774563: Epoch   2 Batch 1430/3125   train_loss = 1.270\n",
      "2017-12-16T13:54:05.935400: Epoch   2 Batch 1450/3125   train_loss = 1.233\n",
      "2017-12-16T13:54:08.643593: Epoch   2 Batch 1470/3125   train_loss = 1.095\n",
      "2017-12-16T13:54:11.058076: Epoch   2 Batch 1490/3125   train_loss = 1.169\n",
      "2017-12-16T13:54:13.019068: Epoch   2 Batch 1510/3125   train_loss = 1.054\n",
      "2017-12-16T13:54:14.907646: Epoch   2 Batch 1530/3125   train_loss = 1.364\n",
      "2017-12-16T13:54:16.800086: Epoch   2 Batch 1550/3125   train_loss = 1.036\n",
      "2017-12-16T13:54:18.764119: Epoch   2 Batch 1570/3125   train_loss = 1.057\n",
      "2017-12-16T13:54:20.665407: Epoch   2 Batch 1590/3125   train_loss = 1.148\n",
      "2017-12-16T13:54:22.535894: Epoch   2 Batch 1610/3125   train_loss = 1.039\n",
      "2017-12-16T13:54:24.460022: Epoch   2 Batch 1630/3125   train_loss = 1.064\n",
      "2017-12-16T13:54:26.361281: Epoch   2 Batch 1650/3125   train_loss = 0.965\n",
      "2017-12-16T13:54:28.148216: Epoch   2 Batch 1670/3125   train_loss = 0.967\n",
      "2017-12-16T13:54:30.003488: Epoch   2 Batch 1690/3125   train_loss = 1.095\n",
      "2017-12-16T13:54:31.900151: Epoch   2 Batch 1710/3125   train_loss = 1.160\n",
      "2017-12-16T13:54:33.780928: Epoch   2 Batch 1730/3125   train_loss = 1.185\n",
      "2017-12-16T13:54:35.653185: Epoch   2 Batch 1750/3125   train_loss = 1.037\n",
      "2017-12-16T13:54:37.572951: Epoch   2 Batch 1770/3125   train_loss = 1.213\n",
      "2017-12-16T13:54:39.535433: Epoch   2 Batch 1790/3125   train_loss = 1.270\n",
      "2017-12-16T13:54:41.465823: Epoch   2 Batch 1810/3125   train_loss = 1.159\n",
      "2017-12-16T13:54:43.363865: Epoch   2 Batch 1830/3125   train_loss = 1.131\n",
      "2017-12-16T13:54:45.261787: Epoch   2 Batch 1850/3125   train_loss = 1.155\n",
      "2017-12-16T13:54:47.132020: Epoch   2 Batch 1870/3125   train_loss = 1.205\n",
      "2017-12-16T13:54:49.057572: Epoch   2 Batch 1890/3125   train_loss = 0.989\n",
      "2017-12-16T13:54:51.055514: Epoch   2 Batch 1910/3125   train_loss = 1.094\n",
      "2017-12-16T13:54:52.992317: Epoch   2 Batch 1930/3125   train_loss = 1.146\n",
      "2017-12-16T13:54:54.900127: Epoch   2 Batch 1950/3125   train_loss = 1.041\n",
      "2017-12-16T13:54:56.784607: Epoch   2 Batch 1970/3125   train_loss = 1.130\n",
      "2017-12-16T13:54:58.645900: Epoch   2 Batch 1990/3125   train_loss = 1.064\n",
      "2017-12-16T13:55:00.527133: Epoch   2 Batch 2010/3125   train_loss = 1.011\n",
      "2017-12-16T13:55:02.384921: Epoch   2 Batch 2030/3125   train_loss = 1.147\n",
      "2017-12-16T13:55:04.323916: Epoch   2 Batch 2050/3125   train_loss = 1.111\n",
      "2017-12-16T13:55:06.290090: Epoch   2 Batch 2070/3125   train_loss = 1.148\n",
      "2017-12-16T13:55:08.244183: Epoch   2 Batch 2090/3125   train_loss = 1.099\n",
      "2017-12-16T13:55:10.162583: Epoch   2 Batch 2110/3125   train_loss = 1.256\n",
      "2017-12-16T13:55:12.246512: Epoch   2 Batch 2130/3125   train_loss = 1.089\n",
      "2017-12-16T13:55:14.169626: Epoch   2 Batch 2150/3125   train_loss = 1.273\n",
      "2017-12-16T13:55:16.059883: Epoch   2 Batch 2170/3125   train_loss = 1.088\n",
      "2017-12-16T13:55:18.001884: Epoch   2 Batch 2190/3125   train_loss = 1.161\n",
      "2017-12-16T13:55:19.889326: Epoch   2 Batch 2210/3125   train_loss = 1.080\n",
      "2017-12-16T13:55:21.798111: Epoch   2 Batch 2230/3125   train_loss = 1.121\n",
      "2017-12-16T13:55:23.688989: Epoch   2 Batch 2250/3125   train_loss = 1.206\n",
      "2017-12-16T13:55:25.559098: Epoch   2 Batch 2270/3125   train_loss = 1.058\n",
      "2017-12-16T13:55:27.434581: Epoch   2 Batch 2290/3125   train_loss = 1.019\n",
      "2017-12-16T13:55:29.288812: Epoch   2 Batch 2310/3125   train_loss = 1.039\n",
      "2017-12-16T13:55:31.156614: Epoch   2 Batch 2330/3125   train_loss = 1.117\n",
      "2017-12-16T13:55:32.979674: Epoch   2 Batch 2350/3125   train_loss = 1.261\n",
      "2017-12-16T13:55:35.032675: Epoch   2 Batch 2370/3125   train_loss = 1.160\n",
      "2017-12-16T13:55:38.168725: Epoch   2 Batch 2390/3125   train_loss = 1.147\n",
      "2017-12-16T13:55:40.246481: Epoch   2 Batch 2410/3125   train_loss = 1.204\n",
      "2017-12-16T13:55:42.533174: Epoch   2 Batch 2430/3125   train_loss = 1.149\n",
      "2017-12-16T13:55:45.707068: Epoch   2 Batch 2450/3125   train_loss = 1.221\n",
      "2017-12-16T13:55:48.055276: Epoch   2 Batch 2470/3125   train_loss = 1.205\n",
      "2017-12-16T13:55:50.212188: Epoch   2 Batch 2490/3125   train_loss = 1.124\n",
      "2017-12-16T13:55:52.514157: Epoch   2 Batch 2510/3125   train_loss = 1.213\n",
      "2017-12-16T13:55:55.668799: Epoch   2 Batch 2530/3125   train_loss = 0.983\n",
      "2017-12-16T13:55:59.177068: Epoch   2 Batch 2550/3125   train_loss = 1.265\n",
      "2017-12-16T13:56:01.334236: Epoch   2 Batch 2570/3125   train_loss = 1.191\n",
      "2017-12-16T13:56:03.424174: Epoch   2 Batch 2590/3125   train_loss = 1.179\n",
      "2017-12-16T13:56:06.826970: Epoch   2 Batch 2610/3125   train_loss = 1.310\n",
      "2017-12-16T13:56:08.836024: Epoch   2 Batch 2630/3125   train_loss = 0.912\n",
      "2017-12-16T13:56:10.757800: Epoch   2 Batch 2650/3125   train_loss = 1.125\n",
      "2017-12-16T13:56:12.714707: Epoch   2 Batch 2670/3125   train_loss = 1.158\n",
      "2017-12-16T13:56:15.135466: Epoch   2 Batch 2690/3125   train_loss = 1.190\n",
      "2017-12-16T13:56:17.111799: Epoch   2 Batch 2710/3125   train_loss = 1.152\n",
      "2017-12-16T13:56:19.556101: Epoch   2 Batch 2730/3125   train_loss = 1.261\n",
      "2017-12-16T13:56:21.925919: Epoch   2 Batch 2750/3125   train_loss = 1.144\n",
      "2017-12-16T13:56:25.167927: Epoch   2 Batch 2770/3125   train_loss = 1.055\n",
      "2017-12-16T13:56:28.550802: Epoch   2 Batch 2790/3125   train_loss = 1.034\n",
      "2017-12-16T13:56:30.477035: Epoch   2 Batch 2810/3125   train_loss = 1.192\n",
      "2017-12-16T13:56:32.386767: Epoch   2 Batch 2830/3125   train_loss = 1.034\n",
      "2017-12-16T13:56:34.284843: Epoch   2 Batch 2850/3125   train_loss = 1.316\n",
      "2017-12-16T13:56:36.147729: Epoch   2 Batch 2870/3125   train_loss = 0.975\n",
      "2017-12-16T13:56:38.089275: Epoch   2 Batch 2890/3125   train_loss = 0.937\n",
      "2017-12-16T13:56:40.069872: Epoch   2 Batch 2910/3125   train_loss = 1.184\n",
      "2017-12-16T13:56:42.104602: Epoch   2 Batch 2930/3125   train_loss = 1.067\n",
      "2017-12-16T13:56:44.032237: Epoch   2 Batch 2950/3125   train_loss = 1.298\n",
      "2017-12-16T13:56:45.928494: Epoch   2 Batch 2970/3125   train_loss = 1.079\n",
      "2017-12-16T13:56:47.839473: Epoch   2 Batch 2990/3125   train_loss = 1.103\n",
      "2017-12-16T13:56:49.771679: Epoch   2 Batch 3010/3125   train_loss = 1.054\n",
      "2017-12-16T13:56:51.655965: Epoch   2 Batch 3030/3125   train_loss = 1.057\n",
      "2017-12-16T13:56:53.540416: Epoch   2 Batch 3050/3125   train_loss = 1.133\n",
      "2017-12-16T13:56:55.392837: Epoch   2 Batch 3070/3125   train_loss = 1.182\n",
      "2017-12-16T13:56:57.270023: Epoch   2 Batch 3090/3125   train_loss = 1.064\n",
      "2017-12-16T13:56:59.090257: Epoch   2 Batch 3110/3125   train_loss = 1.041\n",
      "2017-12-16T13:57:00.744513: Epoch   2 Batch   18/781   test_loss = 1.013\n",
      "2017-12-16T13:57:01.034183: Epoch   2 Batch   38/781   test_loss = 1.118\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2017-12-16T13:57:01.369527: Epoch   2 Batch   58/781   test_loss = 1.125\n",
      "2017-12-16T13:57:01.668262: Epoch   2 Batch   78/781   test_loss = 1.100\n",
      "2017-12-16T13:57:01.954058: Epoch   2 Batch   98/781   test_loss = 1.166\n",
      "2017-12-16T13:57:02.272902: Epoch   2 Batch  118/781   test_loss = 1.131\n",
      "2017-12-16T13:57:02.558785: Epoch   2 Batch  138/781   test_loss = 1.288\n",
      "2017-12-16T13:57:02.847151: Epoch   2 Batch  158/781   test_loss = 1.121\n",
      "2017-12-16T13:57:03.137857: Epoch   2 Batch  178/781   test_loss = 1.018\n",
      "2017-12-16T13:57:03.451644: Epoch   2 Batch  198/781   test_loss = 1.244\n",
      "2017-12-16T13:57:03.739493: Epoch   2 Batch  218/781   test_loss = 1.268\n",
      "2017-12-16T13:57:04.021634: Epoch   2 Batch  238/781   test_loss = 1.161\n",
      "2017-12-16T13:57:04.344650: Epoch   2 Batch  258/781   test_loss = 1.277\n",
      "2017-12-16T13:57:04.629948: Epoch   2 Batch  278/781   test_loss = 1.253\n",
      "2017-12-16T13:57:04.913880: Epoch   2 Batch  298/781   test_loss = 1.149\n",
      "2017-12-16T13:57:05.206107: Epoch   2 Batch  318/781   test_loss = 1.151\n",
      "2017-12-16T13:57:05.533449: Epoch   2 Batch  338/781   test_loss = 1.147\n",
      "2017-12-16T13:57:05.823936: Epoch   2 Batch  358/781   test_loss = 1.061\n",
      "2017-12-16T13:57:06.107951: Epoch   2 Batch  378/781   test_loss = 1.080\n",
      "2017-12-16T13:57:06.425163: Epoch   2 Batch  398/781   test_loss = 1.027\n",
      "2017-12-16T13:57:06.716469: Epoch   2 Batch  418/781   test_loss = 1.261\n",
      "2017-12-16T13:57:06.999421: Epoch   2 Batch  438/781   test_loss = 1.308\n",
      "2017-12-16T13:57:07.334104: Epoch   2 Batch  458/781   test_loss = 1.141\n",
      "2017-12-16T13:57:07.625251: Epoch   2 Batch  478/781   test_loss = 1.296\n",
      "2017-12-16T13:57:07.910266: Epoch   2 Batch  498/781   test_loss = 0.991\n",
      "2017-12-16T13:57:08.202455: Epoch   2 Batch  518/781   test_loss = 1.174\n",
      "2017-12-16T13:57:08.530611: Epoch   2 Batch  538/781   test_loss = 1.051\n",
      "2017-12-16T13:57:08.826681: Epoch   2 Batch  558/781   test_loss = 1.104\n",
      "2017-12-16T13:57:09.118463: Epoch   2 Batch  578/781   test_loss = 1.051\n",
      "2017-12-16T13:57:09.452032: Epoch   2 Batch  598/781   test_loss = 1.234\n",
      "2017-12-16T13:57:09.744078: Epoch   2 Batch  618/781   test_loss = 1.066\n",
      "2017-12-16T13:57:10.036715: Epoch   2 Batch  638/781   test_loss = 1.118\n",
      "2017-12-16T13:57:10.360120: Epoch   2 Batch  658/781   test_loss = 1.238\n",
      "2017-12-16T13:57:10.642094: Epoch   2 Batch  678/781   test_loss = 1.115\n",
      "2017-12-16T13:57:10.928571: Epoch   2 Batch  698/781   test_loss = 1.124\n",
      "2017-12-16T13:57:11.253515: Epoch   2 Batch  718/781   test_loss = 1.181\n",
      "2017-12-16T13:57:11.543373: Epoch   2 Batch  738/781   test_loss = 1.086\n",
      "2017-12-16T13:57:11.842894: Epoch   2 Batch  758/781   test_loss = 1.126\n",
      "2017-12-16T13:57:12.118748: Epoch   2 Batch  778/781   test_loss = 1.229\n",
      "2017-12-16T13:57:13.402397: Epoch   3 Batch    5/3125   train_loss = 1.177\n",
      "2017-12-16T13:57:15.272775: Epoch   3 Batch   25/3125   train_loss = 1.158\n",
      "2017-12-16T13:57:17.142562: Epoch   3 Batch   45/3125   train_loss = 1.059\n",
      "2017-12-16T13:57:19.076768: Epoch   3 Batch   65/3125   train_loss = 1.259\n",
      "2017-12-16T13:57:20.975904: Epoch   3 Batch   85/3125   train_loss = 1.103\n",
      "2017-12-16T13:57:22.854052: Epoch   3 Batch  105/3125   train_loss = 1.011\n",
      "2017-12-16T13:57:24.694647: Epoch   3 Batch  125/3125   train_loss = 1.207\n",
      "2017-12-16T13:57:26.555527: Epoch   3 Batch  145/3125   train_loss = 1.242\n",
      "2017-12-16T13:57:28.451137: Epoch   3 Batch  165/3125   train_loss = 1.131\n",
      "2017-12-16T13:57:30.341108: Epoch   3 Batch  185/3125   train_loss = 1.081\n",
      "2017-12-16T13:57:32.198782: Epoch   3 Batch  205/3125   train_loss = 1.015\n",
      "2017-12-16T13:57:34.101001: Epoch   3 Batch  225/3125   train_loss = 0.918\n",
      "2017-12-16T13:57:36.010647: Epoch   3 Batch  245/3125   train_loss = 1.364\n",
      "2017-12-16T13:57:37.912028: Epoch   3 Batch  265/3125   train_loss = 1.055\n",
      "2017-12-16T13:57:39.812253: Epoch   3 Batch  285/3125   train_loss = 1.164\n",
      "2017-12-16T13:57:41.716688: Epoch   3 Batch  305/3125   train_loss = 1.121\n",
      "2017-12-16T13:57:45.075430: Epoch   3 Batch  325/3125   train_loss = 1.121\n",
      "2017-12-16T13:57:47.400881: Epoch   3 Batch  345/3125   train_loss = 1.150\n",
      "2017-12-16T13:57:49.334115: Epoch   3 Batch  365/3125   train_loss = 1.203\n",
      "2017-12-16T13:57:51.207415: Epoch   3 Batch  385/3125   train_loss = 1.121\n",
      "2017-12-16T13:57:53.097790: Epoch   3 Batch  405/3125   train_loss = 1.047\n",
      "2017-12-16T13:57:54.983329: Epoch   3 Batch  425/3125   train_loss = 1.202\n",
      "2017-12-16T13:57:56.878234: Epoch   3 Batch  445/3125   train_loss = 1.142\n",
      "2017-12-16T13:57:58.741503: Epoch   3 Batch  465/3125   train_loss = 1.067\n",
      "2017-12-16T13:58:00.647239: Epoch   3 Batch  485/3125   train_loss = 1.285\n",
      "2017-12-16T13:58:02.549780: Epoch   3 Batch  505/3125   train_loss = 1.023\n",
      "2017-12-16T13:58:04.456560: Epoch   3 Batch  525/3125   train_loss = 1.118\n",
      "2017-12-16T13:58:06.380333: Epoch   3 Batch  545/3125   train_loss = 1.058\n",
      "2017-12-16T13:58:08.298134: Epoch   3 Batch  565/3125   train_loss = 1.296\n",
      "2017-12-16T13:58:10.168139: Epoch   3 Batch  585/3125   train_loss = 1.044\n",
      "2017-12-16T13:58:12.067228: Epoch   3 Batch  605/3125   train_loss = 1.214\n",
      "2017-12-16T13:58:14.097412: Epoch   3 Batch  625/3125   train_loss = 1.096\n",
      "2017-12-16T13:58:15.988871: Epoch   3 Batch  645/3125   train_loss = 1.220\n",
      "2017-12-16T13:58:17.914634: Epoch   3 Batch  665/3125   train_loss = 1.203\n",
      "2017-12-16T13:58:19.911216: Epoch   3 Batch  685/3125   train_loss = 1.130\n",
      "2017-12-16T13:58:21.790094: Epoch   3 Batch  705/3125   train_loss = 1.368\n",
      "2017-12-16T13:58:23.713485: Epoch   3 Batch  725/3125   train_loss = 1.146\n",
      "2017-12-16T13:58:25.578030: Epoch   3 Batch  745/3125   train_loss = 1.184\n",
      "2017-12-16T13:58:27.488861: Epoch   3 Batch  765/3125   train_loss = 1.081\n",
      "2017-12-16T13:58:29.426773: Epoch   3 Batch  785/3125   train_loss = 1.217\n",
      "2017-12-16T13:58:31.360995: Epoch   3 Batch  805/3125   train_loss = 1.087\n",
      "2017-12-16T13:58:33.252144: Epoch   3 Batch  825/3125   train_loss = 1.044\n",
      "2017-12-16T13:58:35.104995: Epoch   3 Batch  845/3125   train_loss = 1.063\n",
      "2017-12-16T13:58:36.970057: Epoch   3 Batch  865/3125   train_loss = 1.174\n",
      "2017-12-16T13:58:38.937901: Epoch   3 Batch  885/3125   train_loss = 1.126\n",
      "2017-12-16T13:58:40.925649: Epoch   3 Batch  905/3125   train_loss = 1.246\n",
      "2017-12-16T13:58:42.810206: Epoch   3 Batch  925/3125   train_loss = 1.080\n",
      "2017-12-16T13:58:44.738405: Epoch   3 Batch  945/3125   train_loss = 1.258\n",
      "2017-12-16T13:58:46.644129: Epoch   3 Batch  965/3125   train_loss = 0.976\n",
      "2017-12-16T13:58:48.532592: Epoch   3 Batch  985/3125   train_loss = 1.198\n",
      "2017-12-16T13:58:50.439522: Epoch   3 Batch 1005/3125   train_loss = 1.109\n",
      "2017-12-16T13:58:52.340214: Epoch   3 Batch 1025/3125   train_loss = 1.133\n",
      "2017-12-16T13:58:54.179431: Epoch   3 Batch 1045/3125   train_loss = 1.299\n",
      "2017-12-16T13:58:56.073041: Epoch   3 Batch 1065/3125   train_loss = 1.093\n",
      "2017-12-16T13:58:57.970949: Epoch   3 Batch 1085/3125   train_loss = 0.944\n",
      "2017-12-16T13:58:59.828892: Epoch   3 Batch 1105/3125   train_loss = 0.953\n",
      "2017-12-16T13:59:01.652107: Epoch   3 Batch 1125/3125   train_loss = 1.033\n",
      "2017-12-16T13:59:03.581001: Epoch   3 Batch 1145/3125   train_loss = 1.196\n",
      "2017-12-16T13:59:05.517136: Epoch   3 Batch 1165/3125   train_loss = 1.265\n",
      "2017-12-16T13:59:07.429403: Epoch   3 Batch 1185/3125   train_loss = 0.982\n",
      "2017-12-16T13:59:09.362840: Epoch   3 Batch 1205/3125   train_loss = 1.087\n",
      "2017-12-16T13:59:11.254553: Epoch   3 Batch 1225/3125   train_loss = 1.171\n",
      "2017-12-16T13:59:13.116794: Epoch   3 Batch 1245/3125   train_loss = 1.297\n",
      "2017-12-16T13:59:15.046455: Epoch   3 Batch 1265/3125   train_loss = 1.079\n",
      "2017-12-16T13:59:16.969394: Epoch   3 Batch 1285/3125   train_loss = 1.180\n",
      "2017-12-16T13:59:18.896083: Epoch   3 Batch 1305/3125   train_loss = 1.008\n",
      "2017-12-16T13:59:20.804620: Epoch   3 Batch 1325/3125   train_loss = 1.054\n",
      "2017-12-16T13:59:22.758623: Epoch   3 Batch 1345/3125   train_loss = 1.078\n",
      "2017-12-16T13:59:24.648803: Epoch   3 Batch 1365/3125   train_loss = 1.003\n",
      "2017-12-16T13:59:26.514463: Epoch   3 Batch 1385/3125   train_loss = 1.057\n",
      "2017-12-16T13:59:28.382615: Epoch   3 Batch 1405/3125   train_loss = 1.088\n",
      "2017-12-16T13:59:30.372127: Epoch   3 Batch 1425/3125   train_loss = 1.154\n",
      "2017-12-16T13:59:32.275704: Epoch   3 Batch 1445/3125   train_loss = 1.283\n",
      "2017-12-16T13:59:34.159972: Epoch   3 Batch 1465/3125   train_loss = 1.064\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2017-12-16T13:59:36.062830: Epoch   3 Batch 1485/3125   train_loss = 1.214\n",
      "2017-12-16T13:59:37.989938: Epoch   3 Batch 1505/3125   train_loss = 0.998\n",
      "2017-12-16T13:59:39.829032: Epoch   3 Batch 1525/3125   train_loss = 1.004\n",
      "2017-12-16T13:59:41.740450: Epoch   3 Batch 1545/3125   train_loss = 0.999\n",
      "2017-12-16T13:59:43.574373: Epoch   3 Batch 1565/3125   train_loss = 1.171\n",
      "2017-12-16T13:59:45.511951: Epoch   3 Batch 1585/3125   train_loss = 1.159\n",
      "2017-12-16T13:59:47.439572: Epoch   3 Batch 1605/3125   train_loss = 1.050\n",
      "2017-12-16T13:59:49.306158: Epoch   3 Batch 1625/3125   train_loss = 1.182\n",
      "2017-12-16T13:59:51.161802: Epoch   3 Batch 1645/3125   train_loss = 1.178\n",
      "2017-12-16T13:59:53.106959: Epoch   3 Batch 1665/3125   train_loss = 1.139\n",
      "2017-12-16T13:59:54.990885: Epoch   3 Batch 1685/3125   train_loss = 1.167\n",
      "2017-12-16T13:59:56.893071: Epoch   3 Batch 1705/3125   train_loss = 1.052\n",
      "2017-12-16T13:59:58.806131: Epoch   3 Batch 1725/3125   train_loss = 1.109\n",
      "2017-12-16T14:00:00.702002: Epoch   3 Batch 1745/3125   train_loss = 1.000\n",
      "2017-12-16T14:00:02.592917: Epoch   3 Batch 1765/3125   train_loss = 1.022\n",
      "2017-12-16T14:00:04.466337: Epoch   3 Batch 1785/3125   train_loss = 1.170\n",
      "2017-12-16T14:00:06.419069: Epoch   3 Batch 1805/3125   train_loss = 1.192\n",
      "2017-12-16T14:00:08.338956: Epoch   3 Batch 1825/3125   train_loss = 1.224\n",
      "2017-12-16T14:00:10.200465: Epoch   3 Batch 1845/3125   train_loss = 1.175\n",
      "2017-12-16T14:00:12.106457: Epoch   3 Batch 1865/3125   train_loss = 1.062\n",
      "2017-12-16T14:00:14.034379: Epoch   3 Batch 1885/3125   train_loss = 1.171\n",
      "2017-12-16T14:00:15.956521: Epoch   3 Batch 1905/3125   train_loss = 1.012\n",
      "2017-12-16T14:00:17.816870: Epoch   3 Batch 1925/3125   train_loss = 1.055\n",
      "2017-12-16T14:00:19.731383: Epoch   3 Batch 1945/3125   train_loss = 1.329\n",
      "2017-12-16T14:00:21.709578: Epoch   3 Batch 1965/3125   train_loss = 1.140\n",
      "2017-12-16T14:00:23.562436: Epoch   3 Batch 1985/3125   train_loss = 1.082\n",
      "2017-12-16T14:00:25.443515: Epoch   3 Batch 2005/3125   train_loss = 1.201\n",
      "2017-12-16T14:00:27.297616: Epoch   3 Batch 2025/3125   train_loss = 1.097\n",
      "2017-12-16T14:00:29.146127: Epoch   3 Batch 2045/3125   train_loss = 1.027\n",
      "2017-12-16T14:00:31.004292: Epoch   3 Batch 2065/3125   train_loss = 1.076\n",
      "2017-12-16T14:00:32.930146: Epoch   3 Batch 2085/3125   train_loss = 1.413\n",
      "2017-12-16T14:00:34.897173: Epoch   3 Batch 2105/3125   train_loss = 1.030\n",
      "2017-12-16T14:00:36.797822: Epoch   3 Batch 2125/3125   train_loss = 1.258\n",
      "2017-12-16T14:00:38.732677: Epoch   3 Batch 2145/3125   train_loss = 1.114\n",
      "2017-12-16T14:00:40.721094: Epoch   3 Batch 2165/3125   train_loss = 1.097\n",
      "2017-12-16T14:00:42.986397: Epoch   3 Batch 2185/3125   train_loss = 1.187\n",
      "2017-12-16T14:00:45.933564: Epoch   3 Batch 2205/3125   train_loss = 1.222\n",
      "2017-12-16T14:00:48.033633: Epoch   3 Batch 2225/3125   train_loss = 1.075\n",
      "2017-12-16T14:00:49.975115: Epoch   3 Batch 2245/3125   train_loss = 0.988\n",
      "2017-12-16T14:00:51.882634: Epoch   3 Batch 2265/3125   train_loss = 1.199\n",
      "2017-12-16T14:00:53.762276: Epoch   3 Batch 2285/3125   train_loss = 1.307\n",
      "2017-12-16T14:00:55.654301: Epoch   3 Batch 2305/3125   train_loss = 1.011\n",
      "2017-12-16T14:00:57.544373: Epoch   3 Batch 2325/3125   train_loss = 1.107\n",
      "2017-12-16T14:00:59.481457: Epoch   3 Batch 2345/3125   train_loss = 1.077\n",
      "2017-12-16T14:01:01.372526: Epoch   3 Batch 2365/3125   train_loss = 1.003\n",
      "2017-12-16T14:01:03.270500: Epoch   3 Batch 2385/3125   train_loss = 1.086\n",
      "2017-12-16T14:01:05.175146: Epoch   3 Batch 2405/3125   train_loss = 1.078\n",
      "2017-12-16T14:01:07.061415: Epoch   3 Batch 2425/3125   train_loss = 1.042\n",
      "2017-12-16T14:01:08.968754: Epoch   3 Batch 2445/3125   train_loss = 1.091\n",
      "2017-12-16T14:01:10.911829: Epoch   3 Batch 2465/3125   train_loss = 1.069\n",
      "2017-12-16T14:01:12.828464: Epoch   3 Batch 2485/3125   train_loss = 0.979\n",
      "2017-12-16T14:01:14.756199: Epoch   3 Batch 2505/3125   train_loss = 1.102\n",
      "2017-12-16T14:01:16.661846: Epoch   3 Batch 2525/3125   train_loss = 1.119\n",
      "2017-12-16T14:01:18.590703: Epoch   3 Batch 2545/3125   train_loss = 1.233\n",
      "2017-12-16T14:01:20.474861: Epoch   3 Batch 2565/3125   train_loss = 1.046\n",
      "2017-12-16T14:01:22.355192: Epoch   3 Batch 2585/3125   train_loss = 1.025\n",
      "2017-12-16T14:01:24.192712: Epoch   3 Batch 2605/3125   train_loss = 1.018\n",
      "2017-12-16T14:01:26.070740: Epoch   3 Batch 2625/3125   train_loss = 1.252\n",
      "2017-12-16T14:01:27.983308: Epoch   3 Batch 2645/3125   train_loss = 1.087\n",
      "2017-12-16T14:01:29.869361: Epoch   3 Batch 2665/3125   train_loss = 1.174\n",
      "2017-12-16T14:01:31.746773: Epoch   3 Batch 2685/3125   train_loss = 1.174\n",
      "2017-12-16T14:01:33.643413: Epoch   3 Batch 2705/3125   train_loss = 0.980\n",
      "2017-12-16T14:01:35.568365: Epoch   3 Batch 2725/3125   train_loss = 1.156\n",
      "2017-12-16T14:01:37.483795: Epoch   3 Batch 2745/3125   train_loss = 1.228\n",
      "2017-12-16T14:01:39.447936: Epoch   3 Batch 2765/3125   train_loss = 1.062\n",
      "2017-12-16T14:01:41.376217: Epoch   3 Batch 2785/3125   train_loss = 1.332\n",
      "2017-12-16T14:01:43.278789: Epoch   3 Batch 2805/3125   train_loss = 1.120\n",
      "2017-12-16T14:01:45.178934: Epoch   3 Batch 2825/3125   train_loss = 1.156\n",
      "2017-12-16T14:01:47.103341: Epoch   3 Batch 2845/3125   train_loss = 1.168\n",
      "2017-12-16T14:01:49.032593: Epoch   3 Batch 2865/3125   train_loss = 1.010\n",
      "2017-12-16T14:01:50.960624: Epoch   3 Batch 2885/3125   train_loss = 1.157\n",
      "2017-12-16T14:01:52.850519: Epoch   3 Batch 2905/3125   train_loss = 0.945\n",
      "2017-12-16T14:01:54.737905: Epoch   3 Batch 2925/3125   train_loss = 1.086\n",
      "2017-12-16T14:01:56.635458: Epoch   3 Batch 2945/3125   train_loss = 1.358\n",
      "2017-12-16T14:01:58.550406: Epoch   3 Batch 2965/3125   train_loss = 1.288\n",
      "2017-12-16T14:02:00.444592: Epoch   3 Batch 2985/3125   train_loss = 1.053\n",
      "2017-12-16T14:02:02.299588: Epoch   3 Batch 3005/3125   train_loss = 0.971\n",
      "2017-12-16T14:02:04.343299: Epoch   3 Batch 3025/3125   train_loss = 1.176\n",
      "2017-12-16T14:02:06.219139: Epoch   3 Batch 3045/3125   train_loss = 1.273\n",
      "2017-12-16T14:02:08.111368: Epoch   3 Batch 3065/3125   train_loss = 1.056\n",
      "2017-12-16T14:02:10.007051: Epoch   3 Batch 3085/3125   train_loss = 1.107\n",
      "2017-12-16T14:02:11.940758: Epoch   3 Batch 3105/3125   train_loss = 1.231\n",
      "2017-12-16T14:02:14.028914: Epoch   3 Batch   17/781   test_loss = 1.188\n",
      "2017-12-16T14:02:14.364500: Epoch   3 Batch   37/781   test_loss = 1.199\n",
      "2017-12-16T14:02:14.660448: Epoch   3 Batch   57/781   test_loss = 1.181\n",
      "2017-12-16T14:02:14.962199: Epoch   3 Batch   77/781   test_loss = 1.179\n",
      "2017-12-16T14:02:15.295450: Epoch   3 Batch   97/781   test_loss = 1.038\n",
      "2017-12-16T14:02:15.586419: Epoch   3 Batch  117/781   test_loss = 1.387\n",
      "2017-12-16T14:02:15.884983: Epoch   3 Batch  137/781   test_loss = 1.167\n",
      "2017-12-16T14:02:16.181621: Epoch   3 Batch  157/781   test_loss = 1.273\n",
      "2017-12-16T14:02:16.529922: Epoch   3 Batch  177/781   test_loss = 1.111\n",
      "2017-12-16T14:02:16.830978: Epoch   3 Batch  197/781   test_loss = 1.149\n",
      "2017-12-16T14:02:17.131019: Epoch   3 Batch  217/781   test_loss = 1.034\n",
      "2017-12-16T14:02:17.470613: Epoch   3 Batch  237/781   test_loss = 1.018\n",
      "2017-12-16T14:02:17.765399: Epoch   3 Batch  257/781   test_loss = 1.310\n",
      "2017-12-16T14:02:18.059762: Epoch   3 Batch  277/781   test_loss = 1.134\n",
      "2017-12-16T14:02:18.411277: Epoch   3 Batch  297/781   test_loss = 1.255\n",
      "2017-12-16T14:02:18.705775: Epoch   3 Batch  317/781   test_loss = 1.311\n",
      "2017-12-16T14:02:18.989507: Epoch   3 Batch  337/781   test_loss = 1.164\n",
      "2017-12-16T14:02:19.319290: Epoch   3 Batch  357/781   test_loss = 1.259\n",
      "2017-12-16T14:02:19.611186: Epoch   3 Batch  377/781   test_loss = 1.151\n",
      "2017-12-16T14:02:19.904204: Epoch   3 Batch  397/781   test_loss = 1.041\n",
      "2017-12-16T14:02:20.190297: Epoch   3 Batch  417/781   test_loss = 1.148\n",
      "2017-12-16T14:02:20.525796: Epoch   3 Batch  437/781   test_loss = 1.055\n",
      "2017-12-16T14:02:20.824708: Epoch   3 Batch  457/781   test_loss = 0.905\n",
      "2017-12-16T14:02:21.105556: Epoch   3 Batch  477/781   test_loss = 1.086\n",
      "2017-12-16T14:02:21.440195: Epoch   3 Batch  497/781   test_loss = 1.037\n",
      "2017-12-16T14:02:21.736950: Epoch   3 Batch  517/781   test_loss = 1.143\n",
      "2017-12-16T14:02:22.026860: Epoch   3 Batch  537/781   test_loss = 1.155\n",
      "2017-12-16T14:02:22.356117: Epoch   3 Batch  557/781   test_loss = 1.165\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2017-12-16T14:02:22.659387: Epoch   3 Batch  577/781   test_loss = 1.101\n",
      "2017-12-16T14:02:22.972409: Epoch   3 Batch  597/781   test_loss = 1.105\n",
      "2017-12-16T14:02:23.304991: Epoch   3 Batch  617/781   test_loss = 1.084\n",
      "2017-12-16T14:02:23.593466: Epoch   3 Batch  637/781   test_loss = 1.000\n",
      "2017-12-16T14:02:23.888795: Epoch   3 Batch  657/781   test_loss = 1.504\n",
      "2017-12-16T14:02:24.178824: Epoch   3 Batch  677/781   test_loss = 1.029\n",
      "2017-12-16T14:02:24.511463: Epoch   3 Batch  697/781   test_loss = 1.194\n",
      "2017-12-16T14:02:24.797912: Epoch   3 Batch  717/781   test_loss = 1.008\n",
      "2017-12-16T14:02:25.080049: Epoch   3 Batch  737/781   test_loss = 0.957\n",
      "2017-12-16T14:02:25.417155: Epoch   3 Batch  757/781   test_loss = 1.385\n",
      "2017-12-16T14:02:25.715510: Epoch   3 Batch  777/781   test_loss = 1.308\n",
      "2017-12-16T14:02:26.519589: Epoch   4 Batch    0/3125   train_loss = 1.294\n",
      "2017-12-16T14:02:28.390284: Epoch   4 Batch   20/3125   train_loss = 1.022\n",
      "2017-12-16T14:02:30.231071: Epoch   4 Batch   40/3125   train_loss = 1.152\n",
      "2017-12-16T14:02:32.068848: Epoch   4 Batch   60/3125   train_loss = 0.964\n",
      "2017-12-16T14:02:33.919649: Epoch   4 Batch   80/3125   train_loss = 1.044\n",
      "2017-12-16T14:02:35.747065: Epoch   4 Batch  100/3125   train_loss = 1.072\n",
      "2017-12-16T14:02:37.659307: Epoch   4 Batch  120/3125   train_loss = 1.232\n",
      "2017-12-16T14:02:39.639995: Epoch   4 Batch  140/3125   train_loss = 1.204\n",
      "2017-12-16T14:02:41.589785: Epoch   4 Batch  160/3125   train_loss = 0.925\n",
      "2017-12-16T14:02:43.521195: Epoch   4 Batch  180/3125   train_loss = 1.043\n",
      "2017-12-16T14:02:45.444541: Epoch   4 Batch  200/3125   train_loss = 1.289\n",
      "2017-12-16T14:02:47.305874: Epoch   4 Batch  220/3125   train_loss = 1.191\n",
      "2017-12-16T14:02:49.207292: Epoch   4 Batch  240/3125   train_loss = 1.128\n",
      "2017-12-16T14:02:51.137647: Epoch   4 Batch  260/3125   train_loss = 1.123\n",
      "2017-12-16T14:02:53.010572: Epoch   4 Batch  280/3125   train_loss = 1.262\n",
      "2017-12-16T14:02:54.868883: Epoch   4 Batch  300/3125   train_loss = 1.222\n",
      "2017-12-16T14:02:56.736931: Epoch   4 Batch  320/3125   train_loss = 1.198\n",
      "2017-12-16T14:02:58.619563: Epoch   4 Batch  340/3125   train_loss = 0.950\n",
      "2017-12-16T14:03:00.530943: Epoch   4 Batch  360/3125   train_loss = 1.163\n",
      "2017-12-16T14:03:02.434208: Epoch   4 Batch  380/3125   train_loss = 1.060\n",
      "2017-12-16T14:03:04.354510: Epoch   4 Batch  400/3125   train_loss = 0.983\n",
      "2017-12-16T14:03:06.266927: Epoch   4 Batch  420/3125   train_loss = 1.072\n",
      "2017-12-16T14:03:08.136331: Epoch   4 Batch  440/3125   train_loss = 1.105\n",
      "2017-12-16T14:03:10.013445: Epoch   4 Batch  460/3125   train_loss = 1.078\n",
      "2017-12-16T14:03:11.922616: Epoch   4 Batch  480/3125   train_loss = 1.256\n",
      "2017-12-16T14:03:13.865919: Epoch   4 Batch  500/3125   train_loss = 0.874\n",
      "2017-12-16T14:03:15.757218: Epoch   4 Batch  520/3125   train_loss = 1.109\n",
      "2017-12-16T14:03:17.646684: Epoch   4 Batch  540/3125   train_loss = 1.001\n",
      "2017-12-16T14:03:19.577966: Epoch   4 Batch  560/3125   train_loss = 1.117\n",
      "2017-12-16T14:03:21.508829: Epoch   4 Batch  580/3125   train_loss = 1.215\n",
      "2017-12-16T14:03:23.410238: Epoch   4 Batch  600/3125   train_loss = 1.191\n",
      "2017-12-16T14:03:25.293793: Epoch   4 Batch  620/3125   train_loss = 1.191\n",
      "2017-12-16T14:03:27.113852: Epoch   4 Batch  640/3125   train_loss = 1.046\n",
      "2017-12-16T14:03:29.000334: Epoch   4 Batch  660/3125   train_loss = 1.131\n",
      "2017-12-16T14:03:30.888956: Epoch   4 Batch  680/3125   train_loss = 1.032\n",
      "2017-12-16T14:03:32.735442: Epoch   4 Batch  700/3125   train_loss = 1.144\n",
      "2017-12-16T14:03:34.775325: Epoch   4 Batch  720/3125   train_loss = 1.039\n",
      "2017-12-16T14:03:36.694167: Epoch   4 Batch  740/3125   train_loss = 1.185\n",
      "2017-12-16T14:03:38.563852: Epoch   4 Batch  760/3125   train_loss = 1.112\n",
      "2017-12-16T14:03:40.478152: Epoch   4 Batch  780/3125   train_loss = 1.237\n",
      "2017-12-16T14:03:42.440512: Epoch   4 Batch  800/3125   train_loss = 1.097\n",
      "2017-12-16T14:03:44.358635: Epoch   4 Batch  820/3125   train_loss = 1.114\n",
      "2017-12-16T14:03:46.248041: Epoch   4 Batch  840/3125   train_loss = 1.071\n",
      "2017-12-16T14:03:48.540807: Epoch   4 Batch  860/3125   train_loss = 1.009\n",
      "2017-12-16T14:03:51.520494: Epoch   4 Batch  880/3125   train_loss = 1.107\n",
      "2017-12-16T14:03:53.567997: Epoch   4 Batch  900/3125   train_loss = 1.074\n",
      "2017-12-16T14:03:55.467853: Epoch   4 Batch  920/3125   train_loss = 1.223\n",
      "2017-12-16T14:03:57.298861: Epoch   4 Batch  940/3125   train_loss = 1.210\n",
      "2017-12-16T14:03:59.091687: Epoch   4 Batch  960/3125   train_loss = 1.153\n",
      "2017-12-16T14:04:00.975973: Epoch   4 Batch  980/3125   train_loss = 1.190\n",
      "2017-12-16T14:04:02.866809: Epoch   4 Batch 1000/3125   train_loss = 1.138\n",
      "2017-12-16T14:04:04.814512: Epoch   4 Batch 1020/3125   train_loss = 1.171\n",
      "2017-12-16T14:04:06.732899: Epoch   4 Batch 1040/3125   train_loss = 1.077\n",
      "2017-12-16T14:04:08.570552: Epoch   4 Batch 1060/3125   train_loss = 1.314\n",
      "2017-12-16T14:04:10.498043: Epoch   4 Batch 1080/3125   train_loss = 1.087\n",
      "2017-12-16T14:04:12.412993: Epoch   4 Batch 1100/3125   train_loss = 1.110\n",
      "2017-12-16T14:04:14.299746: Epoch   4 Batch 1120/3125   train_loss = 1.161\n",
      "2017-12-16T14:04:16.183547: Epoch   4 Batch 1140/3125   train_loss = 1.165\n",
      "2017-12-16T14:04:18.108570: Epoch   4 Batch 1160/3125   train_loss = 1.154\n",
      "2017-12-16T14:04:20.094977: Epoch   4 Batch 1180/3125   train_loss = 1.093\n",
      "2017-12-16T14:04:21.950320: Epoch   4 Batch 1200/3125   train_loss = 1.044\n",
      "2017-12-16T14:04:23.843040: Epoch   4 Batch 1220/3125   train_loss = 1.123\n",
      "2017-12-16T14:04:25.756659: Epoch   4 Batch 1240/3125   train_loss = 1.017\n",
      "2017-12-16T14:04:27.636844: Epoch   4 Batch 1260/3125   train_loss = 1.082\n",
      "2017-12-16T14:04:29.484934: Epoch   4 Batch 1280/3125   train_loss = 1.116\n",
      "2017-12-16T14:04:31.314486: Epoch   4 Batch 1300/3125   train_loss = 1.047\n",
      "2017-12-16T14:04:33.171075: Epoch   4 Batch 1320/3125   train_loss = 1.085\n",
      "2017-12-16T14:04:35.075214: Epoch   4 Batch 1340/3125   train_loss = 0.912\n",
      "2017-12-16T14:04:36.986425: Epoch   4 Batch 1360/3125   train_loss = 1.018\n",
      "2017-12-16T14:04:38.943599: Epoch   4 Batch 1380/3125   train_loss = 1.041\n",
      "2017-12-16T14:04:40.895641: Epoch   4 Batch 1400/3125   train_loss = 1.180\n",
      "2017-12-16T14:04:42.801092: Epoch   4 Batch 1420/3125   train_loss = 1.222\n",
      "2017-12-16T14:04:44.712900: Epoch   4 Batch 1440/3125   train_loss = 0.970\n",
      "2017-12-16T14:04:46.609053: Epoch   4 Batch 1460/3125   train_loss = 1.117\n",
      "2017-12-16T14:04:48.520892: Epoch   4 Batch 1480/3125   train_loss = 1.082\n",
      "2017-12-16T14:04:50.471666: Epoch   4 Batch 1500/3125   train_loss = 1.162\n",
      "2017-12-16T14:04:52.484065: Epoch   4 Batch 1520/3125   train_loss = 1.077\n",
      "2017-12-16T14:04:54.378484: Epoch   4 Batch 1540/3125   train_loss = 1.167\n",
      "2017-12-16T14:04:56.257897: Epoch   4 Batch 1560/3125   train_loss = 1.100\n",
      "2017-12-16T14:04:58.112882: Epoch   4 Batch 1580/3125   train_loss = 1.185\n",
      "2017-12-16T14:04:59.996210: Epoch   4 Batch 1600/3125   train_loss = 1.104\n",
      "2017-12-16T14:05:01.892474: Epoch   4 Batch 1620/3125   train_loss = 1.178\n",
      "2017-12-16T14:05:03.806359: Epoch   4 Batch 1640/3125   train_loss = 1.151\n",
      "2017-12-16T14:05:05.665100: Epoch   4 Batch 1660/3125   train_loss = 1.171\n",
      "2017-12-16T14:05:07.517155: Epoch   4 Batch 1680/3125   train_loss = 1.154\n",
      "2017-12-16T14:05:09.423520: Epoch   4 Batch 1700/3125   train_loss = 1.036\n",
      "2017-12-16T14:05:11.349973: Epoch   4 Batch 1720/3125   train_loss = 1.021\n",
      "2017-12-16T14:05:13.243031: Epoch   4 Batch 1740/3125   train_loss = 1.001\n",
      "2017-12-16T14:05:15.137378: Epoch   4 Batch 1760/3125   train_loss = 1.220\n",
      "2017-12-16T14:05:17.068574: Epoch   4 Batch 1780/3125   train_loss = 1.008\n",
      "2017-12-16T14:05:19.029400: Epoch   4 Batch 1800/3125   train_loss = 1.080\n",
      "2017-12-16T14:05:20.938504: Epoch   4 Batch 1820/3125   train_loss = 0.984\n",
      "2017-12-16T14:05:22.831303: Epoch   4 Batch 1840/3125   train_loss = 1.134\n",
      "2017-12-16T14:05:24.725150: Epoch   4 Batch 1860/3125   train_loss = 1.157\n",
      "2017-12-16T14:05:26.614325: Epoch   4 Batch 1880/3125   train_loss = 1.070\n",
      "2017-12-16T14:05:28.523117: Epoch   4 Batch 1900/3125   train_loss = 0.993\n",
      "2017-12-16T14:05:30.390655: Epoch   4 Batch 1920/3125   train_loss = 1.123\n",
      "2017-12-16T14:05:32.301955: Epoch   4 Batch 1940/3125   train_loss = 1.051\n",
      "2017-12-16T14:05:34.260412: Epoch   4 Batch 1960/3125   train_loss = 0.970\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2017-12-16T14:05:36.158086: Epoch   4 Batch 1980/3125   train_loss = 1.112\n",
      "2017-12-16T14:05:38.087428: Epoch   4 Batch 2000/3125   train_loss = 1.154\n",
      "2017-12-16T14:05:39.991476: Epoch   4 Batch 2020/3125   train_loss = 1.155\n",
      "2017-12-16T14:05:41.875647: Epoch   4 Batch 2040/3125   train_loss = 0.981\n",
      "2017-12-16T14:05:43.800822: Epoch   4 Batch 2060/3125   train_loss = 0.941\n",
      "2017-12-16T14:05:45.744994: Epoch   4 Batch 2080/3125   train_loss = 1.220\n",
      "2017-12-16T14:05:47.641703: Epoch   4 Batch 2100/3125   train_loss = 0.994\n",
      "2017-12-16T14:05:49.545173: Epoch   4 Batch 2120/3125   train_loss = 1.019\n",
      "2017-12-16T14:05:51.418799: Epoch   4 Batch 2140/3125   train_loss = 1.043\n",
      "2017-12-16T14:05:53.303550: Epoch   4 Batch 2160/3125   train_loss = 1.083\n",
      "2017-12-16T14:05:55.156055: Epoch   4 Batch 2180/3125   train_loss = 1.127\n",
      "2017-12-16T14:05:57.018538: Epoch   4 Batch 2200/3125   train_loss = 1.098\n",
      "2017-12-16T14:05:58.882278: Epoch   4 Batch 2220/3125   train_loss = 1.031\n",
      "2017-12-16T14:06:00.786181: Epoch   4 Batch 2240/3125   train_loss = 1.005\n",
      "2017-12-16T14:06:02.641042: Epoch   4 Batch 2260/3125   train_loss = 1.162\n",
      "2017-12-16T14:06:04.535551: Epoch   4 Batch 2280/3125   train_loss = 1.099\n",
      "2017-12-16T14:06:06.455204: Epoch   4 Batch 2300/3125   train_loss = 1.110\n",
      "2017-12-16T14:06:08.464728: Epoch   4 Batch 2320/3125   train_loss = 1.264\n",
      "2017-12-16T14:06:10.401985: Epoch   4 Batch 2340/3125   train_loss = 1.115\n",
      "2017-12-16T14:06:12.352107: Epoch   4 Batch 2360/3125   train_loss = 1.168\n",
      "2017-12-16T14:06:14.209810: Epoch   4 Batch 2380/3125   train_loss = 1.146\n",
      "2017-12-16T14:06:16.136459: Epoch   4 Batch 2400/3125   train_loss = 1.121\n",
      "2017-12-16T14:06:18.029709: Epoch   4 Batch 2420/3125   train_loss = 1.013\n",
      "2017-12-16T14:06:19.955697: Epoch   4 Batch 2440/3125   train_loss = 1.152\n",
      "2017-12-16T14:06:21.896479: Epoch   4 Batch 2460/3125   train_loss = 1.093\n",
      "2017-12-16T14:06:23.823758: Epoch   4 Batch 2480/3125   train_loss = 1.172\n",
      "2017-12-16T14:06:25.746913: Epoch   4 Batch 2500/3125   train_loss = 1.100\n",
      "2017-12-16T14:06:27.649885: Epoch   4 Batch 2520/3125   train_loss = 1.034\n",
      "2017-12-16T14:06:29.568508: Epoch   4 Batch 2540/3125   train_loss = 1.037\n",
      "2017-12-16T14:06:31.450385: Epoch   4 Batch 2560/3125   train_loss = 0.862\n",
      "2017-12-16T14:06:33.413543: Epoch   4 Batch 2580/3125   train_loss = 1.118\n",
      "2017-12-16T14:06:35.333424: Epoch   4 Batch 2600/3125   train_loss = 1.053\n",
      "2017-12-16T14:06:37.259092: Epoch   4 Batch 2620/3125   train_loss = 1.036\n",
      "2017-12-16T14:06:39.648237: Epoch   4 Batch 2640/3125   train_loss = 1.028\n",
      "2017-12-16T14:06:42.684986: Epoch   4 Batch 2660/3125   train_loss = 1.179\n",
      "2017-12-16T14:06:44.669599: Epoch   4 Batch 2680/3125   train_loss = 1.049\n",
      "2017-12-16T14:06:46.576440: Epoch   4 Batch 2700/3125   train_loss = 1.176\n",
      "2017-12-16T14:06:48.505025: Epoch   4 Batch 2720/3125   train_loss = 0.997\n",
      "2017-12-16T14:06:50.563458: Epoch   4 Batch 2740/3125   train_loss = 1.112\n",
      "2017-12-16T14:06:52.478397: Epoch   4 Batch 2760/3125   train_loss = 1.025\n",
      "2017-12-16T14:06:54.416007: Epoch   4 Batch 2780/3125   train_loss = 1.137\n",
      "2017-12-16T14:06:56.307526: Epoch   4 Batch 2800/3125   train_loss = 1.223\n",
      "2017-12-16T14:06:58.186014: Epoch   4 Batch 2820/3125   train_loss = 1.284\n",
      "2017-12-16T14:07:00.022868: Epoch   4 Batch 2840/3125   train_loss = 1.123\n",
      "2017-12-16T14:07:01.924997: Epoch   4 Batch 2860/3125   train_loss = 1.039\n",
      "2017-12-16T14:07:03.857257: Epoch   4 Batch 2880/3125   train_loss = 1.057\n",
      "2017-12-16T14:07:05.757707: Epoch   4 Batch 2900/3125   train_loss = 1.115\n",
      "2017-12-16T14:07:07.653819: Epoch   4 Batch 2920/3125   train_loss = 1.037\n",
      "2017-12-16T14:07:09.608767: Epoch   4 Batch 2940/3125   train_loss = 1.090\n",
      "2017-12-16T14:07:11.491085: Epoch   4 Batch 2960/3125   train_loss = 1.054\n",
      "2017-12-16T14:07:13.396020: Epoch   4 Batch 2980/3125   train_loss = 1.077\n",
      "2017-12-16T14:07:15.301853: Epoch   4 Batch 3000/3125   train_loss = 1.117\n",
      "2017-12-16T14:07:17.229465: Epoch   4 Batch 3020/3125   train_loss = 1.119\n",
      "2017-12-16T14:07:19.129344: Epoch   4 Batch 3040/3125   train_loss = 1.127\n",
      "2017-12-16T14:07:21.051293: Epoch   4 Batch 3060/3125   train_loss = 1.050\n",
      "2017-12-16T14:07:22.924718: Epoch   4 Batch 3080/3125   train_loss = 1.243\n",
      "2017-12-16T14:07:24.835132: Epoch   4 Batch 3100/3125   train_loss = 1.129\n",
      "2017-12-16T14:07:26.809564: Epoch   4 Batch 3120/3125   train_loss = 1.080\n",
      "2017-12-16T14:07:27.473946: Epoch   4 Batch   16/781   test_loss = 1.099\n",
      "2017-12-16T14:07:27.771051: Epoch   4 Batch   36/781   test_loss = 1.150\n",
      "2017-12-16T14:07:28.061123: Epoch   4 Batch   56/781   test_loss = 1.149\n",
      "2017-12-16T14:07:28.400768: Epoch   4 Batch   76/781   test_loss = 1.172\n",
      "2017-12-16T14:07:28.697335: Epoch   4 Batch   96/781   test_loss = 1.332\n",
      "2017-12-16T14:07:28.982629: Epoch   4 Batch  116/781   test_loss = 1.036\n",
      "2017-12-16T14:07:29.311234: Epoch   4 Batch  136/781   test_loss = 1.053\n",
      "2017-12-16T14:07:29.609368: Epoch   4 Batch  156/781   test_loss = 1.172\n",
      "2017-12-16T14:07:29.903048: Epoch   4 Batch  176/781   test_loss = 1.158\n",
      "2017-12-16T14:07:30.201765: Epoch   4 Batch  196/781   test_loss = 0.981\n",
      "2017-12-16T14:07:30.533448: Epoch   4 Batch  216/781   test_loss = 1.224\n",
      "2017-12-16T14:07:30.827702: Epoch   4 Batch  236/781   test_loss = 1.033\n",
      "2017-12-16T14:07:31.120492: Epoch   4 Batch  256/781   test_loss = 1.135\n",
      "2017-12-16T14:07:31.444140: Epoch   4 Batch  276/781   test_loss = 1.368\n",
      "2017-12-16T14:07:31.737299: Epoch   4 Batch  296/781   test_loss = 1.130\n",
      "2017-12-16T14:07:32.024731: Epoch   4 Batch  316/781   test_loss = 1.088\n",
      "2017-12-16T14:07:32.363541: Epoch   4 Batch  336/781   test_loss = 1.010\n",
      "2017-12-16T14:07:32.649450: Epoch   4 Batch  356/781   test_loss = 1.109\n",
      "2017-12-16T14:07:32.941371: Epoch   4 Batch  376/781   test_loss = 1.174\n",
      "2017-12-16T14:07:33.250381: Epoch   4 Batch  396/781   test_loss = 1.281\n",
      "2017-12-16T14:07:33.565620: Epoch   4 Batch  416/781   test_loss = 1.107\n",
      "2017-12-16T14:07:33.856150: Epoch   4 Batch  436/781   test_loss = 1.246\n",
      "2017-12-16T14:07:34.143701: Epoch   4 Batch  456/781   test_loss = 1.038\n",
      "2017-12-16T14:07:34.482861: Epoch   4 Batch  476/781   test_loss = 1.312\n",
      "2017-12-16T14:07:34.778767: Epoch   4 Batch  496/781   test_loss = 1.173\n",
      "2017-12-16T14:07:35.063477: Epoch   4 Batch  516/781   test_loss = 1.069\n",
      "2017-12-16T14:07:35.396302: Epoch   4 Batch  536/781   test_loss = 1.360\n",
      "2017-12-16T14:07:35.691051: Epoch   4 Batch  556/781   test_loss = 1.142\n",
      "2017-12-16T14:07:35.989057: Epoch   4 Batch  576/781   test_loss = 1.378\n",
      "2017-12-16T14:07:36.312620: Epoch   4 Batch  596/781   test_loss = 1.202\n",
      "2017-12-16T14:07:36.594430: Epoch   4 Batch  616/781   test_loss = 1.130\n",
      "2017-12-16T14:07:36.893662: Epoch   4 Batch  636/781   test_loss = 1.074\n",
      "2017-12-16T14:07:37.192224: Epoch   4 Batch  656/781   test_loss = 1.242\n",
      "2017-12-16T14:07:37.526371: Epoch   4 Batch  676/781   test_loss = 1.439\n",
      "2017-12-16T14:07:37.818020: Epoch   4 Batch  696/781   test_loss = 1.135\n",
      "2017-12-16T14:07:38.114261: Epoch   4 Batch  716/781   test_loss = 1.127\n",
      "2017-12-16T14:07:38.459543: Epoch   4 Batch  736/781   test_loss = 1.324\n",
      "2017-12-16T14:07:38.763522: Epoch   4 Batch  756/781   test_loss = 1.054\n",
      "2017-12-16T14:07:39.060914: Epoch   4 Batch  776/781   test_loss = 1.014\n",
      "Model Trained and Saved\n"
     ]
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "%config InlineBackend.figure_format = 'retina'\n",
    "import matplotlib.pyplot as plt\n",
    "import time\n",
    "import datetime\n",
    "\n",
    "losses = {'train':[], 'test':[]}\n",
    "\n",
    "with tf.Session(graph=train_graph) as sess:\n",
    "    \n",
    "    #搜集数据给tensorBoard用\n",
    "    # Keep track of gradient values and sparsity\n",
    "    grad_summaries = []\n",
    "    for g, v in gradients:\n",
    "        if g is not None:\n",
    "            grad_hist_summary = tf.summary.histogram(\"{}/grad/hist\".format(v.name.replace(':', '_')), g)\n",
    "            sparsity_summary = tf.summary.scalar(\"{}/grad/sparsity\".format(v.name.replace(':', '_')), tf.nn.zero_fraction(g))\n",
    "            grad_summaries.append(grad_hist_summary)\n",
    "            grad_summaries.append(sparsity_summary)\n",
    "    grad_summaries_merged = tf.summary.merge(grad_summaries)\n",
    "        \n",
    "    # Output directory for models and summaries\n",
    "    timestamp = str(int(time.time()))\n",
    "    out_dir = os.path.abspath(os.path.join(os.path.curdir, \"runs\", timestamp))\n",
    "    print(\"Writing to {}\\n\".format(out_dir))\n",
    "     \n",
    "    # Summaries for loss and accuracy\n",
    "    loss_summary = tf.summary.scalar(\"loss\", loss)\n",
    "\n",
    "    # Train Summaries\n",
    "    train_summary_op = tf.summary.merge([loss_summary, grad_summaries_merged])\n",
    "    train_summary_dir = os.path.join(out_dir, \"summaries\", \"train\")\n",
    "    train_summary_writer = tf.summary.FileWriter(train_summary_dir, sess.graph)\n",
    "\n",
    "    # Inference summaries\n",
    "    inference_summary_op = tf.summary.merge([loss_summary])\n",
    "    inference_summary_dir = os.path.join(out_dir, \"summaries\", \"inference\")\n",
    "    inference_summary_writer = tf.summary.FileWriter(inference_summary_dir, sess.graph)\n",
    "\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    saver = tf.train.Saver()\n",
    "    for epoch_i in range(num_epochs):\n",
    "        \n",
    "        #将数据集分成训练集和测试集，随机种子不固定\n",
    "        train_X,test_X, train_y, test_y = train_test_split(features,  \n",
    "                                                           targets_values,  \n",
    "                                                           test_size = 0.2,  \n",
    "                                                           random_state = 0)  \n",
    "        \n",
    "        train_batches = get_batches(train_X, train_y, batch_size)\n",
    "        test_batches = get_batches(test_X, test_y, batch_size)\n",
    "    \n",
    "        #训练的迭代，保存训练损失\n",
    "        for batch_i in range(len(train_X) // batch_size):\n",
    "            x, y = next(train_batches)\n",
    "\n",
    "            categories = np.zeros([batch_size, 18])\n",
    "            for i in range(batch_size):\n",
    "                categories[i] = x.take(6,1)[i]\n",
    "\n",
    "            titles = np.zeros([batch_size, sentences_size])\n",
    "            for i in range(batch_size):\n",
    "                titles[i] = x.take(5,1)[i]\n",
    "\n",
    "            feed = {\n",
    "                uid: np.reshape(x.take(0,1), [batch_size, 1]),\n",
    "                user_gender: np.reshape(x.take(2,1), [batch_size, 1]),\n",
    "                user_age: np.reshape(x.take(3,1), [batch_size, 1]),\n",
    "                user_job: np.reshape(x.take(4,1), [batch_size, 1]),\n",
    "                movie_id: np.reshape(x.take(1,1), [batch_size, 1]),\n",
    "                movie_categories: categories,  #x.take(6,1)\n",
    "                movie_titles: titles,  #x.take(5,1)\n",
    "                targets: np.reshape(y, [batch_size, 1]),\n",
    "                dropout_keep_prob: dropout_keep, #dropout_keep\n",
    "                lr: learning_rate}\n",
    "\n",
    "            step, train_loss, summaries, _ = sess.run([global_step, loss, train_summary_op, train_op], feed)  #cost\n",
    "            losses['train'].append(train_loss)\n",
    "            train_summary_writer.add_summary(summaries, step)  #\n",
    "            \n",
    "            # Show every <show_every_n_batches> batches\n",
    "            if (epoch_i * (len(train_X) // batch_size) + batch_i) % show_every_n_batches == 0:\n",
    "                time_str = datetime.datetime.now().isoformat()\n",
    "                print('{}: Epoch {:>3} Batch {:>4}/{}   train_loss = {:.3f}'.format(\n",
    "                    time_str,\n",
    "                    epoch_i,\n",
    "                    batch_i,\n",
    "                    (len(train_X) // batch_size),\n",
    "                    train_loss))\n",
    "                \n",
    "        #使用测试数据的迭代\n",
    "        for batch_i  in range(len(test_X) // batch_size):\n",
    "            x, y = next(test_batches)\n",
    "            \n",
    "            categories = np.zeros([batch_size, 18])\n",
    "            for i in range(batch_size):\n",
    "                categories[i] = x.take(6,1)[i]\n",
    "\n",
    "            titles = np.zeros([batch_size, sentences_size])\n",
    "            for i in range(batch_size):\n",
    "                titles[i] = x.take(5,1)[i]\n",
    "\n",
    "            feed = {\n",
    "                uid: np.reshape(x.take(0,1), [batch_size, 1]),\n",
    "                user_gender: np.reshape(x.take(2,1), [batch_size, 1]),\n",
    "                user_age: np.reshape(x.take(3,1), [batch_size, 1]),\n",
    "                user_job: np.reshape(x.take(4,1), [batch_size, 1]),\n",
    "                movie_id: np.reshape(x.take(1,1), [batch_size, 1]),\n",
    "                movie_categories: categories,  #x.take(6,1)\n",
    "                movie_titles: titles,  #x.take(5,1)\n",
    "                targets: np.reshape(y, [batch_size, 1]),\n",
    "                dropout_keep_prob: 1,\n",
    "                lr: learning_rate}\n",
    "            \n",
    "            step, test_loss, summaries = sess.run([global_step, loss, inference_summary_op], feed)  #cost\n",
    "\n",
    "            #保存测试损失\n",
    "            losses['test'].append(test_loss)\n",
    "            inference_summary_writer.add_summary(summaries, step)  #\n",
    "\n",
    "            time_str = datetime.datetime.now().isoformat()\n",
    "            if (epoch_i * (len(test_X) // batch_size) + batch_i) % show_every_n_batches == 0:\n",
    "                print('{}: Epoch {:>3} Batch {:>4}/{}   test_loss = {:.3f}'.format(\n",
    "                    time_str,\n",
    "                    epoch_i,\n",
    "                    batch_i,\n",
    "                    (len(test_X) // batch_size),\n",
    "                    test_loss))\n",
    "\n",
    "    # Save Model\n",
    "    saver.save(sess, save_dir)  #, global_step=epoch_i\n",
    "    print('Model Trained and Saved')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 在 TensorBoard 中查看可视化结果"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "tensorboard --logdir /PATH_TO_CODE/runs/1513402825/summaries/\n",
    "\n",
    "<img src=\"assets/loss.png\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 保存参数\n",
    "保存`save_dir` 在生成预测时使用。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "save_params((save_dir))\n",
    "\n",
    "load_dir = load_params()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 显示训练Loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAvIAAAH0CAYAAABfKsnMAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAWJQAAFiUBSVIk8AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzs3Xd8FHX+x/H3NwkJkIQaivQizQII\nClJUwIoiciqeYq/nnV2x/QS7h2c5G3qeqICnd4qgoBTBAgpIDb13JEBogRBIz35/fyRZsmkEmNnJ\nhtfz8eCxuzOzs58EF9/znc98x1hrBQAAACC0hHldAAAAAIBjR5AHAAAAQhBBHgAAAAhBBHkAAAAg\nBBHkAQAAgBBEkAcAAABCEEEeAAAACEEEeQAAACAEEeQBAACAEESQBwAAAEIQQR4AAAAIQQR5AAAA\nIAQR5AEAAIAQRJAHAAAAQhBBHgAAAAhBBHkAAAAgBEV4XYAbjDGbJVWTtMXjUgAAAFCxNZN00Frb\nPNgfXCGDvKRqVapUqdWuXbtaXhcCAACAimv16tVKS0vz5LMrapDf0q5du1rx8fFe1wEAAIAKrHPn\nzlq0aNEWLz6bHnkAAAAgBBHkAQAAgBBEkAcAAABCEEEeAAAACEEEeQAAACAEEeQBAACAEESQBwAA\nAEJQRZ1HHgAAlIHP51NSUpJSUlKUkZEha63XJQGeMcYoKipKsbGxqlWrlsLCyveYN0EeAICTlM/n\n07Zt25Samup1KUC5YK1Venq60tPTdfjwYTVu3Lhch3mCPAAAJ6mkpCSlpqYqIiJC9evXV3R0dLkO\nLYDbfD6fDh8+rMTERKWmpiopKUlxcXFel1Uivq0AAJykUlJSJEn169dXbGwsIR4nvbCwMMXGxqp+\n/fqSjnxHyiu+sQAAnKQyMjIkSdHR0R5XApQv+d+J/O9IeUWQBwDgJJV/YSsj8UAgY4wklfuLv/nm\nAgAAAAXkB/nyjiAPAAAAhCBmrXGItVY5viOnXyLCOUYCAACAe0ibDknP8unUZ6bo1Gem6Iznp3pd\nDgAACCGHDh2SMUb9+vU74X2dffbZiomJcaAq5wwfPlzGGI0dO9brUioUgjwAADhpGWOO6c+oUaO8\nLhnwo7UGAACctJ577rkiy95++20lJyfroYceUo0aNQLWdezY0ZU6oqOjtXr1akdG0seNG1fup02E\nMwjyLijnMxUBAIA8zz//fJFlo0aNUnJysh5++GE1a9YsKHUYY9S2bVtH9tW0aVNH9oPyj9Yah4TI\nLEUAAMAB+X3oaWlpGjJkiE499VRFRkbq/vvvlyTt27dPr776qi644AI1aNBAkZGRqlevnq655hot\nWrSoyP5K6pEfPHiwjDFauHChvvjiC3Xu3FlVqlRRXFycbr75Zu3evbvE2gqaOHGijDF64403NH/+\nfF166aWqXr26YmJidNFFFyk+Pr7Yn/OPP/7QTTfdpLi4OFWtWlWdO3fWV199FbC/EzVnzhxdddVV\niouLU1RUlFq0aKGHH35Ye/bsKbLtjh079NBDD6l169aqWrWqatasqXbt2unOO+/Utm3b/Nv5fD6N\nGDFCXbt2VVxcnKpUqaImTZro8ssv1/jx40+45vKCEXkAAIDj4PP51K9fP61du1aXXnqpateu7R8N\nX7x4sZ577jn16tVLV111lapXr67Nmzfru+++08SJE/Xjjz/q/PPPL/Nnvfbaa5o4caKuuuoq9e7d\nW7Nnz9bnn3+uFStWaOHChQoPDy/TfmbNmqUhQ4aoV69euvvuu7Vp0yaNHz9evXr10ooVKwJG8xMS\nEtStWzft2LFDF154oc455xxt375dt956q/r27Xtsv6wSjBkzRjfeeKPCw8M1cOBANWrUSHPnztU7\n77yjCRMmaPbs2WrQoIEk6eDBg+ratat27NihSy65RAMGDFBWVpa2bt2qsWPH6uabb1bjxo0lSQ8/\n/LDee+89tWrVSjfccINiYmK0Y8cOzZs3T+PHj9eAAQMcqd9rjgR5Y8y1ki6Q1FFSB0mxkr6w1t5U\nzLatJF0t6VJJrSTVk7Rf0lxJb1trpztRk5forAEAoOJLS0tTSkqKVqxYUaSXvlOnTkpMTFTNmjUD\nlm/cuFFdu3bVY489pgULFpT5s37++WctWbJErVu3lpQ77fWAAQP03XffaerUqbr88svLtJ8JEybo\n66+/1rXXXutf9uabb2rw4MF6//339dprr/mXP/bYY9qxY4defPFFDR061L/8b3/7m3r27Fnm2kuS\nlJSku+66S8YYzZo1S2effbZ/3dChQ/Xyyy/r/vvv1zfffCNJmjRpkhISEjRkyBC99NJLAftKT09X\ndna2pCOj8S1bttTy5csVFRUVsO3evXtPuPbywqkR+SHKDfCHJCVIKq3J6yVJf5a0StJkSUmS2kjq\nL6m/MeYha+27DtUFAACOU7OnJnldQpltefUKTz532LBhRUK8JNWqVavY7Vu2bKn+/ftr5MiR2rdv\nn2rXrl2mz3n88cf9IV7K7am/66679N1332n+/PllDvKXXnppQIiXpHvuuUeDBw/W/Pnz/ctSUlL0\nzTffqG7dunr88ccDtj/33HM1cOBAffnll2X6zJJ8/fXXSklJ0d133x0Q4iXpmWee0ccff6wJEyZo\n7969iouL86+rUqVKkX1Vrlw54LUxRpGRkcWeqSi4r1DnVI/8I5JaS6om6a9H2fYHSZ2stadba/9i\nrX3aWnu1pAslZUl63RhzikN1AQAAuKZLly4lrps+fbquvvpqNWrUSJGRkf4pLEeOHCkpt9+7rAoH\nXUn+NpL9+/ef0H5iY2NVvXr1gP2sWLFC2dnZ6ty5c5GQLMmREfn8awX69OlTZF3lypXVvXt3+Xw+\nLV26VJJ08cUXq06dOho6dKj69eun999/X0uWLJHP5wt4b1hYmK6//nqtXr1aZ5xxhoYOHapp06Yp\nJSXlhGsubxwZkS/YDmOOctWntXZUCct/NcbMkHSxpO6SxjlRmyforQEAoMKrWrWqYmNji133+eef\n65ZbblFMTIwuvvhiNW/eXNHR0TLGaNq0aZozZ84xTRFZ3Kh/RERujMvJyTmh/eTvq+B+kpOTJUn1\n6tUrdvuSlh+L/M845ZTix2/zlx84cEBS7kj6vHnz9Pzzz2vixImaNGmSv5YHH3xQTz75pH8E/t//\n/rfatm2r0aNH6+WXX5YkVapUSf3799ebb75ZYWb2KW8Xu2blPWZ7WsVxYNYaAEBF41W7SqgobfBy\nyJAhio2N1eLFi9WiRYuAdevXr9ecOXPcLu+EVKtWTZK0a9euYteXtPxYVK9eXZKUmJhY7PqdO3cG\nbCdJzZs31+jRo+Xz+bRixQr9/PPPGj58uJ555hmFh4frySeflJQb2p944gk98cQTSkxM1MyZM/X5\n559r3LhxWrNmjZYuXVrmC4TLs3Iz/aQxpqly22tSJf1WxvfEF/dHpffoAwAAuCY7O1tbt25Vx44d\ni4T4rKysch/iJenMM89URESE4uPjlZ6eXmT9rFmzTvgzzjrrLEnSjBkziqzLyMjQnDlzZIwp9iZc\nYWFhat++vR555BFNnDhRkkqcVrJ+/foaOHCgJkyYoC5dumjlypXasGHDCddfHpSLIG+MiZL0haQo\nSc9ba8ve7AUAAFCOREREqGHDhlq5cmXADCk+n09PP/20Nm/e7GF1ZRMbG6sBAwZo9+7dev311wPW\nzZs3T19//fUJf8Z1112nmJgYjRw50t8Hn2/YsGHauXOnf355SVqyZIkSEhKK7Cf/7EDVqlUl5c7J\n/+uvvxbZLiMjw9/OU9wFs6HI89YaY0y4pP9I6iHpK0llvrOAtbZzCfuMl9TJkQKPg6VJHgCAk9oj\njzyiwYMHq3379rr66qsVFhamX3/9VVu2bFHfvn01ZcoUr0s8qjfffFOzZs3Ss88+q99++03nnHOO\nEhISNGbMGF155ZUaP368wsKOf0y4Vq1a+uijj3TzzTerW7duGjhwoBo2bKi5c+dq+vTpaty4sYYP\nH+7ffuLEiXruuefUs2dPtWnTRnFxcdq6dasmTJig8PBwDR48WFJuT32vXr3UsmVLdenSRU2aNFFq\naqp++OEHrV+/XoMGDVKTJk1O+PdTHnga5PNC/OeSBkoaI+kma21IpmAjmuQBAECuRx99VDExMRo+\nfLg+/fRTRUdHq1evXhozZoxGjBgREkG+SZMmmjt3rp5++mlNnTpVs2bN0mmnnabRo0crLS1N48eP\n9/fSH68bbrhBTZo00auvvqqJEycqJSVFDRo00AMPPKAhQ4aobt26/m379++vPXv2aObMmfrmm290\n6NAhnXLKKbryyiv12GOP+WfkqV27tv7+979r+vTpmjlzpvbs2aNq1aqpVatWevLJJ3XrrbeeUM3l\niXE6NxtjekmarhJuCFVguwhJ/1VuiP+vpFustWW/7Lr0GuI7derUqaTbDbshM9un1kNyv5SVwo3W\nv1K2+VwBAPDK6tWrJUnt2rXzuBKEmoceekjvvvuuZs2apR49enhdjivK+v3o3LmzFi1atKikThE3\nedIjb4yJlDRWuSH+M0k3OxXiy4PQPKcAAAAQqLi57hcsWKCPPvpIDRo0UNeuXT2oCvmC3lqTd2Hr\nN5Iul/SJpHustb7S31X+Mf0kAACoaNq1a6dOnTrp9NNPV+XKlbV27Vp/W9D777/vn8se3nDkt2+M\nGSBpQN7L+nmP3Ywxo/Ke77XWDs57/qFyQ/xeSdslPVvMPKwzrLUznKgNAAAAx+dvf/ubJk+erC++\n+EKHDh1SzZo11a9fPz3xxBPq3r271+Wd9Jw6jOooqfCVAy3y/kjSVkn5Qb553mOcpGdL2ecMh2oL\nOjprAABARTBs2DANGzbM6zJQAkeCvLX2eUnPl3HbXk58ZnlDZw0AAACCqVzcEAoAAADAsSHIAwAA\nAAWEym2NCPIuCJW/fADAyS1/sgmfL+QnjwMclZ/lipmQpVwhyDukvP9FAwBQWFRUlCTp8OHDHlcC\nlC/534n870h5RZAHAOAkFRsbK0lKTExUSkqKfD4fZ5Vx0rLWyufzKSUlRYmJiZKOfEfKK2bxdwH/\nBAIAQkGtWrV0+PBhpaamKiEhwetygHKlatWqqlWrltdllIog7xAaawAAoSYsLEyNGzdWUlKSUlJS\nlJGRwYg8TmrGGEVFRSk2Nla1atVSWFj5bl4hyAMAcBILCwtTXFyc4uLivC4FwDEq34cZIYrBDAAA\nALiNIO8QJq0BAABAMBHkAQAAgBBEkAcAAABCEEEeAAAACEEEeYdwZ1cAAAAEE0EeAAAACEEEeZdw\nQw0AAAC4iSAPAAAAhCCCPAAAABCCCPIuobMGAAAAbiLIO4iJawAAABAsBHkAAAAgBBHkAQAAgBBE\nkHcJLfIAAABwE0HeQbTIAwAAIFgI8gAAAEAIIsi7hDu7AgAAwE0EeQcZ5p8EAABAkBDkAQAAgBBE\nkHcJjTUAAABwE0HeQTTWAAAAIFgI8gAAAEAIIsgDAAAAIYgg7xJmnwQAAICbCPIOYvZJAAAABAtB\nHgAAAAhBBHmXWCagBAAAgIsI8g4yTEAJAACAICHIAwAAACGIIO8SZq0BAACAmwjyTqKzBgAAAEFC\nkAcAAABCEEEeAAAACEEEeQAAACAEEeQdRIs8AAAAgoUgDwAAAIQggrxLmH4SAAAAbiLIO8jQWwMA\nAIAgIcgDAAAAIYgg7xIremsAAADgHoK8gwzz1gAAACBICPIAAABACCLIAwAAACGIIO8Spp8EAACA\nmxwJ8saYa40x7xljZhpjDhpjrDHm86O8p7sxZrIxJskYk2qMWWaMedgYE+5ETV5g+kkAAAAES4RD\n+xkiqYOkQ5ISJLUtbWNjzFWSxklKl/SVpCRJV0p6S1IPSQMdqgsAAACokJxqrXlEUmtJ1ST9tbQN\njTHVJI2QlCOpl7X2Tmvt45I6Spoj6VpjzPUO1eUZOmsAAADgJkeCvLV2urV2vbVl6gy/VlIdSV9a\naxcW2Ee6ckf2paMcDJRXdNYAAAAgWLy42LVP3uMPxaz7TVKqpO7GmKjglQQAAACEFqd65I9Fm7zH\ndYVXWGuzjTGbJZ0uqYWk1aXtyBgTX8KqUnv0g6FsJycAAACA4+PFiHz1vMfkEtbnL68RhFocZZi2\nBgAAAEHixYj80eSn4aMOaVtrOxe7g9yR+k5OFgUAAACUJ16MyOePuFcvYX21QtsBAAAAKMSLIL82\n77F14RXGmAhJzSVlS9oUzKKcRoc8AAAA3ORFkP8l7/GyYtadL6mqpN+ttRnBK8kZdMgDAAAgWLwI\n8mMl7ZV0vTHm7PyFxpjKkl7Oe/kvD+oCAAAAQoYjF7saYwZIGpD3sn7eYzdjzKi853uttYMlyVp7\n0Bhzt3ID/QxjzJeSkiT1V+7UlGMlfeVEXV5i9kkAAAC4yalZazpKurXQshZ5fyRpq6TB+SusteON\nMRdIekbSNZIqS9og6VFJ75bxDrHlD701AAAACBJHgry19nlJzx/je2ZLutyJzwcAAABONl70yJ8c\nQvOcAgAAAEIEQd5BdNYAAAAgWAjyAAAAQAgiyLvE0lsDAAAAFxHkAQAAgBBEkHeQMXTJAwAAIDgI\n8gAAAEAIIsi7JERvaQUAAIAQQZB3EJ01AAAACBaCPAAAABCCCPIuobMGAAAAbiLIO4jOGgAAAAQL\nQR4AAAAIQQR5l1imrQEAAICLCPIO4oZQAAAACBaCPAAAABCCCPIAAABACCLIu4QOeQAAALiJIO8g\nOuQBAAAQLAR5AAAAIAQR5F3C7JMAAABwE0HeQcw+CQAAgGAhyAMAAAAhiCDvEsu8NQAAAHARQd5R\n9NYAAAAgOAjyAAAAQAgiyAMAAAAhiCDvFlrkAQAA4CKCvIOYfhIAAADBQpAHAAAAQhBB3iV01gAA\nAMBNBHkH0VkDAACAYCHIAwAAACGIIO8SS28NAAAAXESQdxCz1gAAACBYCPIAAABACCLIAwAAACGI\nIO8SywSUAAAAcBFB3kGGCSgBAAAQJAR5AAAAIAQR5F3C9JMAAABwE0HeQUw/CQAAgGAhyAMAAAAh\niCDvEjprAAAA4CaCvIPorAEAAECwEORdYrnaFQAAAC4iyDvIcLUrAAAAgoQg7xIG5AEAAOAmgjwA\nAAAQggjyAAAAQAgiyDuIFnkAAAAEC0HeJfTIAwAAwE0EeQcxIg8AAIBgIci7xHJvVwAAALjI0yBv\njLnCGDPNGJNgjEkzxmwyxnxtjOnmZV3Hy3BvVwAAAASJZ0HeGPMPSRMldZL0g6R3JC2SdJWk2caY\nm7yqzQn0yAMAAMBNEV58qDGmvqTBknZJam+t3V1gXW9Jv0h6UdLnXtR3vOiRBwAAQLB4NSLfNO+z\n5xUM8ZJkrZ0uKUVSHS8KcwoD8gAAAHCTV0F+vaRMSV2MMXEFVxhjzpcUK+knLwo7EQzIAwAAIFg8\naa2x1iYZY56U9E9Jq4wx4yXtk9RSUn9JP0r6y9H2Y4yJL2FVW6dqPV6WJnkAAAC4yJMgL0nW2reN\nMVskfSrp7gKrNkgaVbjlJhSYAk3yxHgAAAC4yctZa56QNFbSKOWOxEdL6ixpk6QvjDGvHW0f1trO\nxf2RtMbF0ktEaw0AAACCxZMgb4zpJekfkr6z1j5qrd1krU211i6S9CdJ2yU9Zoxp4UV9TqCzBgAA\nAG7yakS+X97j9MIrrLWpkuYrt7azglnUCWNIHgAAAEHiVZCPynssaYrJ/OWZQajFJQzJAwAAwD1e\nBfmZeY/3GGMaFlxhjOkrqYekdEm/B7uwE8GAPAAAAILFq1lrxip3nviLJK02xnwrKVFSO+W23RhJ\nT1lr93lU3wmjRx4AAABu8moeeZ8x5nJJ90m6XrkXuFaVlCRpsqR3rbXTvKjtRBScfhIAAABwk5fz\nyGdJejvvT4XDgDwAAADc5Nk88hUR4/EAAAAIFoK8S+iRBwAAgJsI8g6iRR4AAADBQpB3iaVLHgAA\nAC4iyDvI0CUPAACAICHIu4QeeQAAALiJIO8geuQBAAAQLAR5lzAiDwAAADcR5AEAAIAQRJB3CbPW\nAAAAwE0EeQeZAk3ytNYAAADATQR5B3GtKwAAAIKFIA8AAACEIIK8g5h+EgAAAMFCkHcJPfIAAABw\nE0HeQYzIAwAAIFgI8i5h+kkAAAC4iSDvIMO8NQAAAAgSgrxL6JEHAACAmwjyDqJHHgAAAMFCkHcJ\nA/IAAABwE0HeQQzIAwAAIFgI8i6xNMkDAADARQR5J9EkDwAAgCAhyLuE8XgAAAC4iSDvIMbjAQAA\nECwEeZfQIg8AAAA3EeQdRIs8AAAAgoUg7xqG5AEAAOAegryDCg7I01oDAAAANxHkHWTorQEAAECQ\nEORdwoA8AAAA3ESQdxDj8QAAAAgWgrxL6JEHAACAmwjyDqJFHgAAAMFCkHeJZUgeAAAALiLIO8jQ\nJQ8AAIAgIci7hPF4AAAAuIkg7yQG5AEAABAkBHmX0CIPAAAANxHkHcSAPAAAAIKFIO8SS5c8AAAA\nXESQdxDzyAMAACBYCPJuYUAeAAAALiLIO4h55AEAABAsBHmXMCAPAAAANxHkHUSPPAAAAIKFIO8S\n5pEHAACAmwjyDmJEHgAAAMFCkHcJ88gDAADATQR5BxWctYbWGgAAALiJIO8gWmsAAAAQLAR5lzAg\nDwAAADd5HuSNMecZY8YZY3YaYzLyHqcZYy73ujYAAACgvIrw8sONMUMkvSRpr6SJknZKipN0lqRe\nkiZ7VtwJsjTJAwAAwEWeBXljzEDlhvifJF1trU0ptL6SJ4WdAEOTPAAAAILEk9YaY0yYpH9ISpU0\nqHCIlyRrbVbQC3MQ4/EAAABwk1cj8t0lNZc0VtJ+Y8wVks6QlC5pvrV2jkd1nRDG4wEAABAsXgX5\nc/Ied0laJOnMgiuNMb9JutZau6e0nRhj4ktY1faEKzxRDMkDAADARV7NWlM37/FeSVUkXSQpVrmj\n8lMlnS/pa29KO360yAMAACBYvBqRD897NModeV+a93qlMeZPktZJusAY0620NhtrbefilueN1Hdy\nsuBjZRmSBwAAgIu8GpHfn/e4qUCIlyRZa9OUOyovSV2CWtUJYkAeAAAAweJVkF+b93ighPX5Qb9K\nEGpxBdPIAwAAwE1eBfnfJGVLamWMiSxm/Rl5j1uCVpEDmEceAAAAweJJkLfW7pX0laTqkp4tuM4Y\nc7GkSyUlS/oh+NU5gxF5AAAAuMmzO7tKelRSV0nPGGPOlzRfUlNJf5KUI+lua21JrTflEuPxAAAA\nCBbPgry1drcxpqukIcoN7+dKSpE0SdIwa+1cr2pzAgPyAAAAcJOXI/Ky1iYpd2T+US/rcAot8gAA\nAAgWry52rfAsTfIAAABwEUHeUUeG5InxAAAAcBNB3kG01gAAACBYCPIuobMGAAAAbiLIO4gBeQAA\nAAQLQd5Ba3el+J+nZmZ7WAkAAAAqOoK8g7buS/U//+eP6zysBAAAABUdQd4lCfvTvC4BAAAAFRhB\nHgAAAAhBBHkAAAAgBBHkAQAAgBBEkAcAAABCEEEeAAAACEEEeQAAACAEEeQBAACAEESQBwAAAEIQ\nQR4AAAAIQQR5AAAAIAQR5AEAAIAQRJAHAAAAQhBBHgAAAAhBBHkHhYcZ//M7ezb3sBIAAABUdAR5\nB93YtYn/ecMaVTysBAAAABUdQd5BYcYcfSMAAADAAQR5B+09lOF/vmXfYQ8rAQAAQEVHkHfQxGU7\n/c8/m7PVw0oAAABQ0RHkAQAAgBBEkAcAAABCEEEeAAAACEEEeQAAACAEEeQBAACAEESQBwAAAEIQ\nQR4AAAAIQQR5AAAAIAQR5AEAAIAQRJAHAAAAQhBBHgAAAAhBBHkAAAAgBBHkAQAAgBBEkAcAAABC\nEEEeAAAACEEEeQAAACAEEeQBAACAEESQBwAAAEIQQR4AAAAIQQR5AAAAIAQR5AEAAIAQRJAHAAAA\nQhBBHgAAAAhBBHkAAAAgBBHkHXTP+S38z09vUM3DSgAAAFDREeQddGbD6v7nzWpHe1gJAAAAKjqC\nvIOMOfLcynpXCAAAACq8chPkjTE3G2Ns3p+7vK7neBiZo28EAAAAOKBcBHljTGNJ70k65HUtTrEM\nyAMAAMBFngd5Y4yRNFLSPkkfelzOCQlorSHIAwAAwEWeB3lJD0rqI+l2SYc9ruWEFGysoUceAAAA\nbvI0yBtj2kl6VdI71trfvKzFCYzIAwAAIFg8C/LGmAhJ/5H0h6T/86oOZx1J8uR4AAAAuCnCw89+\nVtJZknpaa9OOZwfGmPgSVrU97qpOACPyAAAACBZPRuSNMV2UOwr/prV2jhc1uCFw8kmSPAAAANwT\n9BH5Ai016yQNPZF9WWs7l/AZ8ZI6nci+j4cpMCTPiDwAAADc5MWIfIyk1pLaSUovcBMoK+m5vG1G\n5C1724P6jhu3gwIAAECweNEjnyHpkxLWdVJu3/wsSWslhWzbDQPyAAAAcFPQg3zeha13FbfOGPO8\ncoP8aGvtx8GsywmBF7sS5QEAAOCe8nBDqAojIMh7VwYAAABOAgR5BxlxsSsAAACCo1wFeWvt89Za\nE4ptNZICrnYlxwMAAMBN5SrIh7qCs9bQIw8AAAA3EeQdVHAeeQAAAMBNBHkHFYzxM9fv9awOAAAA\nVHwEeQdt25/qdQkAAAA4SRDkHbRl72GvSwAAAMBJgiDvoLAweuQBAAAQHAR5B4VxsSsAAACChCDv\noHCCPAAAAIKEIO8gWmsAAAAQLAR5B5HjAQAAECwEeQfRWgMAAIBgIcg7iNYaAAAABAtB3kHMWgMA\nAIBgIcgDAAAAIYggDwAAAIQggryD6KwBAABAsBDkHUSOBwAAQLAQ5B3EiDwAAACChSDvIGatAQAA\nQLAQ5B0UzjzyAAAACBKCvIOa1q7qdQkAAAA4SRDkHdSleW2vSwAAAMBJgiDvkqqR4V6XAAAAgAqM\nIO+ggh3yqZk5ntUBAACAio8g76LE5HSvSwAAAEAFRZB3UOHZJ1PSs7wpBAAAABUeQd5FTCsPAAAA\ntxDkHWQUmNyt9agQAAAAVHgEeRf5CPIAAABwCUHeQYVbaaxI8gAAAHAHQd5BhVtpsnMI8gAAAHAH\nQd5BvkJJ/r/z//CoEgAAAFQ68kiTAAAgAElEQVR0BHkHFQ7y6xJTPKoEAAAAFR1B3kFVKoUHvC4c\n7AEAAACnEOQdFBEe+OuMCOPXCwAAAHeQNF00qGsTr0sAAABABUWQd9gFrev4n0dHRXhYCQAAACoy\ngrzDwsOOTCaflpXjYSUAAACoyAjyDvtlzW7/809mbvKwEgAAAFRkBHkXLU1I9roEAAAAVFAEeQAA\nACAEEeQBAACAEESQBwAAAEIQQR4AAAAIQQR5AAAAIAQR5AEAAIAQRJAHAAAAQhBBHgAAAAhBBHkA\nAAAgBBHkAQAAgBBEkHfZht0pXpcAAACACogg77COjWsEvD6QmuVRJQAAAKjICPIOa1izSsDrHJ/1\nqBIAAABUZAR5h93YpUnAa3I8AAAA3OBJkDfG1DbG3GWM+dYYs8EYk2aMSTbGzDLG3GmMCdkDjFPr\nxgS83p+a6VElAAAAqMi8CswDJY2Q1FXSPElvSxon6QxJH0saY4wxHtV2QiLCA3+lf/tikUeVAAAA\noCKL8Ohz10nqL2mStdaXv9AY83+S5ku6RtLVyg33IaVWdKTXJQAAAOAk4MmIvLX2F2vt9wVDfN7y\nREkf5r3sFfTCXOKjUR4AAAAOK4+96PnzNWZ7WoWD/vXrRq9LAAAAQAXjVWtNsYwxEZJuyXv5Qxm2\njy9hVVvHinLA61PX6r7ep3pdBgAAACqQ8jYi/6pyL3idbK2d6nUxAAAAQHlVbkbkjTEPSnpM0hpJ\nN5flPdbaziXsK15SJ+eqO3G7U9JVN7ay12UAAACggigXI/LGmPskvSNplaTe1tokj0tyXJdXfta+\nQxlelwEAAIAKwvMgb4x5WNJwSSuUG+ITPS7JNS9OXOV1CQAAAKggPA3yxpgnJb0laYlyQ/xuL+tx\n24QlO2Tt8U1FuScl47jfCwAAgIrHsyBvjBmq3Itb4yVdaK3d61UtwTR3U5J2p6Trns8W6pGvligt\nM+eo73l96hqd88pPuvuzhUGoEAAAAKHAk4tdjTG3SnpRUo6kmZIeNMYU3myLtXZUkEtzRI2qlXQg\nNavYdU9/s0w1qkZqybYDkqQGNSrr8UtLny3z/em589D/tHq3tu47rKa1o50tGAAAACHHq1lrmuc9\nhkt6uIRtfpU0KijVOKxf+1P0+dw/SlyfH+Il6dtF248a5As6nHH0EXwAAABUfJ601lhrn7fWmqP8\n6eVFbU74y/kty7xtZs6x9b376JMHAACAysGsNRVRtcqVSly3ZV9qwOu9TEkJAACA40CQd0ORdv/S\nZef4yrwtA/IV18+rd+nZCSu0ac8hr0sBAAAhgCDvgupVSh6RL07rIVM0a/2RSXs+mLFBd45aoNU7\nDxbZtmBrzeGMbP198mr944c1Ss8qvXc+PStHPh9HAeXVnpQM3Tl6oT6bs1W3jVzgdTkAACAEeHWx\nKwrwWemmT+Zp9B1ddOun8/3LF287oPghFwVsmx/FD2dk6/TnpvqXx0RF6L7epxa7/4VbknTHqAWq\nFR2p7x7oWWrrD7yx6I/9/ud/JKWWsiVCnbVW1kphYcd46g4AgEIYkS9HCoZ4SUo6nKnP5mwNWJY/\nIv+vGRsDlr/3y/rc9YVG3bclperaD+foYHq2tuxL1XMTVkqSRs7erN5vzNCX84ufXScrx1dqy8/B\n9Cy99sMa/fvXjcrxWf2wYqfGLNymzOyytwltP5Cm+75YpBe/X6WcAnX/uGqXHhuzVCu2J5d5X6GO\nSHdyOJiepf7DZ+v816drTWLRM24AABwLRuRd0rpejNbtOvFe5+e+WxnwevKynfrXjI36cdWugOXp\nWT4Nm7JaX8z9Qw9d2Ep3n99CkvT42KUB2327eLteveZMvfD9KknSU98s1/VdmkiScnxW8zbv09j4\nBH2zaLsk6bfHe6tJ7apF6nrrx3UaOXuLJGnYlDUF6sjRLd2alelnGzxmqeZs2idJalM/Rted3VgH\n07L9N776bul2TX7wPP2yZrf6dWighjWqlGm/XknYn6qoiHDViY065veGFb2PAiqgN6au1fK8A9Q7\nRy3U7Kf6eFwR3GCtVfzW/apXrbIa1yr67ycAOIUReZe0O6WaK/v9eNbmIiE+379/3aRDGdl6ZfJq\n/7K5m5KKbLe+0AHGo18tkbVWz05YoUEj5vlDvCQ99vUSrdierJ9W7QoY7c8P8YU9mzfiv/tguiYs\n2a6Hvlysx79eqpT0ojfIyg/xkvTkuOU6d9jPunP0kf7wrByrgf+eo2FT1uievHC/dd9hDRoxV498\ntUS7D6brjlELdOeoBTqQmllsPcfD57NFzkZYa7Vqx8Fiz1KkpGfpkrd+Vc9/TNc5r/ykSct2ylqr\nPm/OULOnJumOUUfveXcix69JPKgnxy4r8b+Pwqy1Wrkjudi/m2CxJ9nV2/Fbj7RQbT+Q5mElcNOX\nC7bp2g/nqNcbM7SDv+cKKzUzW1OW79SeFGafg3cYkXdJuMcjrLM37NWEJduLXdfvvVkBr79ZvF39\nOpyiL+YVbbNZsGW/f/tbuzXVXee10P6jhOaM7Bz1fWem9h0+sl31KpU0pN9ppb5v18EM7ToY+A9i\n/h1yV+7IbUN46Msl/htqfbv4yM/3yqTVen1gB/9ra63mbNqnjCyfLmhdR8ZI4xZtV2Jymm7p3qzE\n6wT2HcrQtR/O0eGMbH1869lq36iGJOmJscv0dXyCzm1RS1/e0y3gc24buSDg7Mt9/12kd67vqE17\nDkuSflmzW98v3aErOzTwb7NpzyG99sNaZfusXru2falBPjUzW5OW7dSZjaqrbf1qysz2KTktK2Dk\n31qry96eKUn6auE2LRp6sWpWraRnJ6zUih3JerH/GTqzUfWA/X702yYNm7JGtaMjNfupPqpcKbzI\nZ6/Ynqzhv2xQrzZ1dN3ZjTV74141rllVzeKi/Z8rScXcmblY+f3hczfv06AR8yRJE+7roQ6Na5Tp\n/V7z+ewJ9bZz5uXk8PQ3yyXlnuV8ZfJqvT+ok8cVwQ2Pf71Mk5bvVJNaVTV9cC+Fc90LPECQd0lU\nJW9Pdtz48bxj2v6OUQuPus3oOVs1ulDPfnGmrdwVEOIlaeTvW3Rvr5b614yNiggzGnh242OqL1/B\nu+IW9HV8QkCQH7douwZ/ndtW9P6gTjqQlqlnvl0hSdq457De+nNHZef4FBEepsnLd2psfIIGdWmi\n75bu0Oa9uQG8//DZGnn7Oerdpq6+jk+QlHuGY2dymk6pXkXxW5N0zb/mFFvPQ18uCXj9wP8W+4P8\nQ18u1oQlO/zrXvh+pQac1bDY/fy+ca8/8ErSnKf76JoPfteulAy99eeO6p+3z6fGLQ9435rEgzqY\nlq3/zM39+7py+CxtefUK/3prrb8lat/hTP133h+6o2dzFTbwwzlKy8rRDysT9VReOIkIM5r9VB9F\nRYTp5k/ma9XOgzqrcQ1d07mRbshr0yoox5fbZlA3Nkp3fbZQaZk5AaPRN4yYq95t62rj7kN6/doO\nOrNRdS36Y7/+NWOjLj29vq7t3KjY382J2HUwXe/9sl7Nakfrzp7NAw5EMrJzFBVR9KBm6spEPTF2\nmZLTsnRb92a6pVtTNaxZRff/d7ESk9P15nUd1LpebJH3WWv9+z+WHJ+V41OYMUXCwYe/btTEZTv0\nyEWtdWG7erLWat7mJNWoWklt6xc9E/jDikR9MW+rbuzaVJedUb/sBTggOS1LicnpalO/6O8l1KRm\nZisyPEwR4cf2b3tqRrZjNRzOyFZ0FP/bLi8mLd8pKXeCghXbk0NmQAIVC/8iuKRFXIzXJXjmgf8t\nLrIsx2d19ss/+V//+7dNx7zfrKPMt38oI1vRkeGatmqXP8RLuSPkBX27eLva1I/VqwV6+6XckfPC\nbh+5ICAAS1J2jlVaZk6JIb4kmdk+7U/NDAjxkjRhyQ6d3bRmwLLHxizVxj2Hihy4dBv2i//5g/9b\nrP4dGshaq68WbgvYbuOew/qw0AXRB9OzVK1yJb390zq9/dP6gHWpmcWHjbRipjXN9lm9MXWtwozx\n93sv3LpfC7fu15kNq+uMhoEj/898u1xfLthWZD9HPjtHk5bl/g9x0Ii5Wv7Cpbr6g98l5V74fH6r\nOO1OyVCT2lWLPZOy91CGqleppErhYUrPytH4xdtVOyZKF59Wr8TPfHLcMs1Yu0eSFBkRplu6NZO1\nVneMWqDfN+7TywPOKHKw+Zf/xPufj/p9iyYv36kbujTxtzJd8tZvRQ6Wpq/drae/Wa7TG1TXx7ec\nXeaLmtfvStGNH89TVKUwjb23u+pVqywptx0n/7/bO0cv1JZXr9C3i7fr0TFLFWakSQ+eF9DWZ63V\nvZ/n1j1z/d4i/y3n27TnkH5dt0fnNKulFyeu0vzNSXqgz6l67JI2Ss3M1rhF29WsdlWd3bSWqkQW\nPcgpLDE5XRHhJuA7v/bly4o9QCrpwMkJOT6rJdsO6PQG1Yo941Scbxcn6H/ztun2Hs3Up11dRUWE\na8GWJN0+coGqVY7QpAfPU83oyDLXUNqsv+lZOfpp9S6d2bC6mtaO9i+31uqzOVu1IzlNf72gpWpU\njdTo37fopYmrdF6rOH162zlFzoJtS0rVul0pOr91HVUqw8FGelaONu05rHanxJb5jFpJUjOzFWZM\nmX/H5UX+YE6+DbsP6bM5W9SrTR31aVvyvx/FyTlKm2BKepZiyzBj3KRlOzVp+Q7d2bO5OjetdUw1\n4OREkHfJrd2bafyS7f6WEJy4Vs9MKXX9GQWm4zyawiG+NM2emhTw+rzXpqvtcYwwth5Scv1DJwRe\n1DxuUUKZ9vnbuj3+0/gB+xu/osiyHq/+osY1q2pVMfcn+N/8bXpj2jq1rR+rr+7ppmpVIkr9n3t6\ntk/LE4qeHflx1S6ddko1Zfl8/nBWWogvLKWY0ctbPp2vNYkpqhRutOy5S/1BMsdn9cL3K/XZnK1q\nWKOKfnr0Ar0/fYOGT98gSRr3127q3LSWDqZnaeeBdKVn5ej1qWvVrWVtf4iXcq/r6Ny0pg6kZml6\n3vLHxy4LCPLjFxdtU9udkqF3fg48IJqwZLuu6thQmdk+XfOv3/0HOrsO7ta4RQlFfqe7DqYrKiJM\nU1cm6sm8syqPXdxa3y7Zrt15fbdDxq/Q033bavLynXpj2roidTwxdpmk3MD46JilmvLQef51hUNk\nwbMD+XJ8VjeMmFukre29Xzbolm7NNOr3zXp/+pGDwrvPa65nrii5TW5cfIIGj11a5OZ17/y0Xj4r\nxUSF694LWioiPEwPf7lYk5cn6unL2+r2HkXPCJWmuJ+lsMfHLtU3i7arTb1Y/fDwef7tl247oEMZ\n2eresnbAPjKzfXrkq9xBgPlbkhQbFaEPbuqk20YuUI7P6lBGtoZNWa3Xru0Q8DlTlu/Up7M3a1DX\nJvrTWYFnkEqLd69OWaNRv29RbOUIzfu/C7UtKU2rdx5UZESYf6KD/Ycz9dq1Hfyvp6/do/mbk9S1\nRW3/fpLTsnTp278pNTNH9/VuqccvbVvi7ywzx6ffN+7T7Xn3q/hbr5Z64rK2ysz2af7mJJ3VpEaZ\nR/1zr7M56G+/HHX7OerVpm6R7TKyc5SWmaMaVct+AHQsMrJztGVvqlrXiynTQUl2jk+DPp6nNTsP\n6p/XdVSftnUVFmZ0yyfztCM5XZ/N2apFQy9WrUIHbD6f1f7UTNWOKTqhQWk5/smxyzQmfpv+cn5L\nPdW3rX9GtpioCD0+dqlqR0fp7es7KivH5x94mrw8sdgD72150xOX5SLqwxnZuvfzeKWkZ+ud6zsG\nHCweK2utliUkq0Wd6DIdkCB4CPIuiYwI08QHeqr505O9LgUuWJOY4nUJknJDblmlpGcXG+KlIxde\nrklMUYcXp0mSTm9Q8gXbPmuL7fd+5+f1/nB7z/kt1L5QX35ZFL4ANv93nZVj1e7ZH9S7TR1FRYTr\nj6RU/8+z/UCaPvx1oz/ESyrxjMmsDXuLLLvi3Vl6c2BgODuUka2YqAj9smaXHv5qSZH3FOehL5eo\nf4cG+nzuVn+IL/hzHCp0oNL91V8Cpl6VpDd/DAzrP67apZXbk7UjOb3I5x3OyFZ2gfev3nnQP8K9\nYXeK7vksvsh7Ctu893CREJ8vOS0zIMRL0oiZm/VHUqqGXd0+IOgs2XZAv67do7d+KnqwIUkfFDhD\nVKNqpM5tUVvj885OvfD9qlKDvLVW63cfUss6MQoPM5qwZLtemrhafc+or5cGnKHMbJ++mLdV1ko3\nndtUkRG5o6z5F+6v3ZWiHq/+otlP9dHy7cm66v3ZkqT3bjgr4NqVwvdwSMnI1s2fBH7HxixM0D+u\nae8PjBt2H9Jfv8gNXwu27NcVZzYI2N5aq/SsHFWuFK4JS7Zr+C8bNKhrE93eo7lG/b4l93PSszVy\n9ha9+/N6ZRSawnfMwoQiBw4J+9PUtcDr/8zZotTM3LNn70/fWGyQT8/K0Z8++L3IjQY/mLFRT1zW\nVo+OWaKJeWfGWtWNUfUqlUpsl5Ok0b9v0fDpGwIu9Lxt5AKteekyZfusoiPDZYzRgdRMXfTP33Qw\nLUsjbj1bF7SuU+z+SpLjswHtZYVH0XN8Vn3fmalNew7rgtZ19Olt5/i3X5N4UHVjKxcJ5GMWJmj+\n5tyJIO76bKGax0Xr63u7BXzHliUcCDgoyfFZXfHuTK3blaKXB5ypQV0L/16KT/Ip6Vn+M6Yf/rpR\nvdrU0fUfzS2y3YjfNqnvmaeU+rtYsu2A/vRB7n+7Y+/trs6FzuTmS8vM0aGMbJ3zypEzYg/+b7Em\n3N9Ti/7Yr3mbknRN54ayVpq2apf6tK1b6qxw6Vk5+mDGRr2b92/7d/f38F8/lu9AaqY+mLFRcTGR\nuqtnC8fvk+HzWX0xb6uS07J0e4/mtJgVwG/CRcYYXX1WQ31TzGgeUN6VdjYpISlVm/KuJSjJR8fR\nPiXpqAe/0wuMphdUeHT8WH1V6MzBGc9NVbPaVbVl37HdoOvFiauKndUpx2e1YfehIsvKorgQLyng\npnD52gz5QafWjSnyWZL0/vQNWp2Yogta1dF15zSWtbbUC/RKGmWcunKXpq78UW3rx+qDGztp+to9\nemniqjL9LJL05rS1+uefOwYsyz/z1atNHVWOCFeLOtFqUz9Wew9l+vd9Xqs4/efOrv5rUP4zd6t6\nt60TcI3PixNXKS4mUo9d0iZg/zuS0zVhyY6Ag72C167sPpiui/75a5nqH/X7Ft3eo7l+37BXgwpd\nj1S4HW3m+r1q/8K03Nmw8v6+X/h+lf/6lXyvT11b4uclpwXOLPXY10t1Td61I3sPZRR7pqawEb9t\nKvZu4fnyQ7wkrc/7b2fh1v3avj9Nt3RvqrqxlZXjs/pp9S71ODWuyNTI+doO/UGS1KV5LX11z7l6\nbepa7T2UG/Zv/XS+zmsVp0cubq2zGtfQ4cwcxRQKZJnZPv+B2P99u1zjF2/X033b6upOjXTffxf5\nz6ate7mvIiPC9Ou63f6JBX5dt0cPfblYwwd10lcL/tCT45arSqVwzX6qjz/MH0zP0v99G3gWc/Pe\nw3p2QuBZzILX8WxLStUNI+YqYX+av67CQT4jy6dFf+ROOXrFuzN1IDVL795wlrq3rB2wXX6rW2E/\nr9ld5Fqp/YczA9q4bvp4nv87+Zf/xGthoRtGSrkHIDd+PE8p6YGDBksTkpWcluVvW/zHD0fOSn9W\nN0bfP9AzoDUqf0Dlpk/maeGW/QEHmP2Hz9aWV6/wD3bk7m+t/pd3X5pf1uxWrzZ1dWu3ZmVqxdu4\n55ASk9PVrUVtGZN7PdLShGT179DA3yo4ZUWi/8z1G9PWac7TfXRK9SralpSq8Yu368J29XRaKYNP\nFZmpiNO/GWPiO3Xq1Ck+/uijUW47kJqpji/+6HUZABDg9AbVtO9Qpq7u1DBgtLygkbef42/BKC+u\naH+K/5oKp7zQ/3T9tHqXZq4veramJNd0alRsC9zioRfrrJfc/zf/pnObqEOjGno8r7WqoGXPX6LE\n5HS1qnuk1aTwRfYFbXn1iiIthIU91bftMbUkHk3VyHClZeWoV+s6urlbU42L3+6/ePStP3dQpyY1\ndcHrM/zb39/71ICDMEkaPugsHc7I9rel5RtyRTu9POnINMw3n9tUz155miqFh+n571b6z4QczXVn\nN9I957fQHaMWFjlbU5bfmSTFxURq76Hjnx55y6tXlJgjimu96fzSj0Umm8g3fNBZuv+/Ra9hyzfk\nina6s2dzXfD6DP2RlKo29WK1dlfJZ5+NkXq1rqNGNasWOTCVpPt6t1Td2MoyRrqhSxNVCg/TmsSD\nSkhKU0zlCHVtXksJ+9N0wevT5bPSywPO0NrElIB9LR56sWpGR+rif/7qP8DM9/39PfX42KVak5ii\niDCj1S9dVqbrQ9zQuXNnLVq0aJG1tnOwP5sgHwRl+bIDAEJf+0bVtSyhfNyV+rLT66tl3WilZfr0\n6ezNJW73j2vOLBKGK6LYqIhir8M5Hl/ec26xLTJOe+Si1iW2q9WrFqU7ezbX3ee1UI7PKiI8rNS8\ncVv3ZmU+iAmGYVefqU9mbS727GFZREaEBdxN/vkrT9Ntx3itjVMI8g4rb0F+1OzNev77sp92BgAA\nKIuuzWtpTWJKkRask9EZDavp3zefHfQ7wXsZ5LmzaxDc1qO5VrxwqWK5OAMAADho3uYkQnyeFdsP\n6pEyTk5QURDkgyQmKkJXtC/9inQAAAAcv/wZiU4WBPkgqoBdTAAAAPAIQT6IbKF5Zkfedo76tT9F\nr1/b3qOKAAAAEKpo2g6iwiPyvdvWVe+2uTecuOS0+v4b8QAAAABHw4h8EJXWWVO9aiVteKWvxt7b\nTWteuqzI+i7NarlXGAAAAEIOQT6IjtYjHxEeprOb1Qq4u1q+Mfd2O6bP+ud1HTThvh7H9B4AAACE\nDoJ8EF1yej3/8zMalv1WwqXNh9o8Llo3n9tU1Sof6ZIa85duurpTI4WZkm+9nu/81nUCXp/XKk6z\nnuytp/q2LfE9wwedVYaqpTb1YgNe332eNzdqAAAAqIjokQ+iS06rp/t7n6qNew6VGpRLcmu3pho9\nJ/fWxdWrVFL7RtU1+vYuCgszemnAGcq/uVf+LblPa1BNDWtU0fYDaf591IqO1KH0bEVVCtO0R85X\nRpZPvd6Y4V//nzu7SpLu6NG82NtxP3lZW/Vr36DIbZ4vO72+Hr64lQ5nZOtgWrY6Nq6hmtGRRd5f\nNTJC7/y8PmDZyNvPUacmNdXhhSPXCFx3diONWVj09uf5FjxzkQ6kZurit34rcZuyqF6lki4/s77+\nN3/bCe0HAAAg2AjyQWSM0eBL25Rp28GXtNYb03Jvy/zEZbnvefbK03X5maeoXYNqio6MUHhY4Ii7\nKTQCHx5m9M3fumvupn3KzPZpW1Kqbu3eTMYYVa4UpqqRuX/9153dSBOW7Ag4uIiMCNPkB8/T5e/O\nDNjnvRe0kJTbuvPomKWSpIcubKVHLm5dpp+rd9u6AUH+b71aqlfrOjLGaPx9PfT2T+v0QJ9T1blp\nLU1ZnhhwO+2oiDBlZPs09eHzVSc2SnVio7Tx75fr9417tWXvYQ2dsNK/beNaVbQtKU3FaVijiq5o\nf4q2JaXqzes6KMwYRUWEH9etq+NiIvX4pW1Ut1pl9WpdR9k+q1bPTAnYZt7/Xah61SorK8entYkp\n6vferGP+nJLc2q2pth9I10+rdx3ze2/s2kRfzPujxPVvDOygwV8vPZHyAnx4Uye98P0q7UxOL/N7\n5jzdR92G/eJYDQAAVCTGVsDJzY0x8Z06deoUHx/vdSnHLT0rR6N/36KqUREa1KVJkdDutMxsnyIj\ninZaJSan69xhP0uSOjetqXF/7e5ft35XiurERqlG1aIj76X5dNZmLd52QA9f1Eot68SUuN2CLUn6\n87/nKMwYTXywp1rWiVGl8OK7wbbsPRxwZuGXxy7QixNXKcwYzVq/V5k5Pv+6mU/0VuNaVYvsw1qr\nycsTdd9/F5Vaf/O4aE156DylZ+WoamREkd9bs6cm+Z/XiY3SgmcuClifdDhTyxIOqMepcQGh/8K2\ndbVw6/5i79B3Q5cm6ti4ulLSs9W+UQ2d3qCaovPuFHzX6AX6afXuUmsuLDIiTOte7qu5m/bp+o/m\nFll/Y9cmeuVPZwb8LJL04lWnKzUzR29OW6sX+p+hQV2b+NetTUzRtJWJurJDA/V+c0aRa0LWv9JX\nEWFGzZ+e7F8284neOu+16cXW2LZ+rKY8dF7uzz9iruZuOr6bfIQZafVLl2n2hr26Y9TC49pHvtu6\nN9OPq3YFnOXKd0X7UzRp2c4T2n9F8K8bO+mvX5T+HSrsvRvO0gP/W3z0DQHgKGpHRyp+6MVB/czO\nnTtr0aJFi6y1nYP6wSLIowymr92teZuSdEu3pmpQSr++G3YcSFNEuFHd2MpH3faTWZv1y5pdevii\n1jqnwCw/8zbt002fzFNkeJh+ePj8YkN8QWMWbNOT3yxT/WqVNX1wLz3y1RJNWZHoX7952OVFzn4U\nVDD81o2N0vxCQb6kbV+7pr3ObFRdfd85chakVd0YPXfl6erZKq7EfdwxaoF+WXMkyNeOjtT9fU7V\nZ3O2qk/buhra7zQt3XZAV70/27/NxAd66oyG1SVJH/22UX+fHNhGte7lvoqMCAuob8J9PdShcY0S\n6yjo0a+W6JvF2/2v373h/9u78/go6vuP469PTkKAhAQQCAHCoYLc9w2CWhURpdSr3if1rv39flYf\n2KL1Z/Gs1vqrtZ6t91XUar3wBhW8xQPkiIIICMgdcs7vj/km7G52k01IdrPh/Xw89rHJzPe7M/uZ\n7+x8ZuY7M4M5emBnwN9hWvNTEV3aZmBm1XYWwH8S8uv/NYn2rdMBKCuv4Iu129iwvZhz/uEn4xdP\n7sW8T9by3eZdYedhzrS+TBvYmayMVFLcDuCqjTs5OGCHrzZ/OKYfr3+9gSP7d+Lwfh1plZ5CaXkF\nW4tKGXbtq1Xlrv95fzYc/30AABsrSURBVI4f3pWSsgqKy8pZ8eNOjnHxvuDgntzx+oqopjeyIIdr\nj+lHb3d9yZLvt3Lrq8uYc/RBjLs+/A5PbQrnTuWChz8Ku5MxtFtbzp3Qg/P+WfNv5ev/NYnO2S14\n/rMfmPufr0ky488nDua4v70bdnqBcR6Un016ShLvR3ja4hVHHMh5E3vy/spNHB9mpzJag7tmM6F3\n+2pd9xrSY+eO2qt5DDV1QCe657aMun00trSUJErKKmovCLx62UQOueXNOn3+bScM4pJHP6nPrEmc\ntEhNYndpdG2iqfj4qkPDdu1tTPFM5NW1Rmp18AEdOPiADnGZdl12HM4aV8BZ46pfUDuyRy7vXjGF\n9JQkWrdIrfVzjhuez7je7chtlUZ6SjJXTz+I+V9toKS8gj9MP6jGJD5UNBccV/KnF3x0/5XLJtZa\n78yxBVWJ/DGDOnPDzIGkpSRxxtg9sRiYn81dpwzl+y1FHD88v6pbFcC5E3pSXFrBza/4XbmOHZxX\ndZbhmQvG8sDCQo7s3ynqJB5g9lF92bSzhArP40/HD6Jdq/SqcWYWtDN1xtju3LegkMy0ZN69cgpb\nd5VW29lKSU5iYH42nudx48wB/LSrhJNHdeOs8T14c9mPjOvVjoUrNnLRIx/jeX6XrV+O6lbtDE5B\nu0wK507lm/XbmfHXhWzfXUZOZhoPnT0yaAcK4MlZoxnWPYdTRnULGp6anBT0fQKlpSSRlpLEoPxs\nHjhzBD/tLGFot7ZBidpzF45j2l+qd6+64ecDOG54ftCwfnlZ3H3a8LDTykhNpmNWC1Zt3Mmxg/OY\nPbUPc577kuc+XVtVZsaQPAD+OKM/6SlJPP3R90Gfcd8Zw0lNCo7Re1dMoWNWCyoqPLYUlZKVkVp1\nRnDGkC7MGNIFz/PCrgd/O8XfhhW0y2TZtUewZVcJHdq0YNvuUgbMqf6cjLSUJM6b2BPw19MlV/+M\ntVuKWPnjDn710Ee0Tk/h9pOGcNq9i8LGAPyL71ukJDOyRw6tW6Ry5tgCfv34J0E7t7U5YXg+n3+/\nlS/WbmNI12xuPX4wf397JVuKSoPiOaIgh4+uOpSbXl5K15yWfLtpZ9X1NbmZaWzaWVLrtDpnteDq\n6f04pE8HzAzP8+iUlcEHhZu5bkZ/+v7upajnu1K/vDb8+6Lx9LzyBcorqh+c+/OJg2mdnsIZ9y8O\nGt4qPYUdAd0XP77qUNJTkli1cSdPfriGtJQkRhTkcMo9wfE/b2IPenVoxauXTeCQW94iLzuDv548\nhKP/soBIvvnfI0hNTuLQvvvx4HvfkmTGtc9/FfV37Jbbkm837eKiyb14Z/lGPv5uS9R1I2mdnhLU\nfbNS4dypYQ8w1EXLtGQGdMni+OH5DOnalq9+2MasB/0zVYf13Y+Xv6x7d8hTR3fjjLEFEQ9G/P3U\nYdzyyjK++mFbjZ/z5KzRHH/Xe2HbSqCrjurLWeMKmH7HAj5d7cf76IGdeTZgnQj16mUTaN0ilXvf\nWcVna7aycUcx32zYUfMXa0CdslrEPImPNx2RF4nC2i1FfL+liGHd2taayAduADplteDdK6ZELPvw\n+9/xu2eWMCg/mydmjcbM+MWdC1lc+BOnju7GNdP71Tpvnufx2OLVrPmpiLPHF9S5qxP4Xbmuff5L\ndhWXM/uovuTE8Idwd2k5L32xjn55WTV2tYpGRYVHUpTd0FZt3Mm/P13L4f060nu/1jz14Rp+464J\nCDxjEUngcq48Ih9OcVk5B8x+ser/wrlTOe7Od1lUuOcI9XXH9uf44fk1dqELTSxevWwiXdpm8MXa\nrQzKb0tykrFpRzE3vbyM5z9by/HD8zl/Uq+gjVrgZ/x+Wt+qnb2v121j0arNTBvQuU4bwSc+WM3l\nT31G15yWvHjphLC3zgUoLa8I6kb2q0k9WbZuO1dO7RNxma/dUkRWRiqZ6Slc9vgnPP3R95w0sit5\n2Rnc+NJSwD/a/6/zx4RdJ0vKKrj+xa+5551VAMwc2oWZQ7tU60r21K/GMLRb27Dz8MPWoqBrNArn\nTg0av3VXKfe8s5IubVtSXFYedJ3O+N7tePubjdU+87kLx9G/S+S2Fbqcx/bKpVNWBoUbd3LdjP70\n7tCKK/+1hEcW7bm+5f9+OYQj+3diV0kZT324Jmg+wD/Dlpps3PTyUu54fQUXT+nNaaO7kdsqndLy\nCr7dtJPO2RlBO/iR5qlfXhseO3d0Vde+QNf++0vudvEOdMdJQ5g6oFO14c9+upabXlrKsYPzyEhL\nDnuDhcDvsH7b7qqd/K1Fpby3clO1s0kHdmzN1+u2V6t/4oh8zhhbwJ9eWcZ/lqyjT6c2PHrOKJ7/\n/AceXfwdn63ZWlW28qzSY4tXM6VPB2579RveWb6RHu0zWbO5KKirZiShbQVg2+5SMlKTWbZ+O1P/\nHPlaqTE9czmyfydmz1sSNPzx80YzoiCHQ255k+VhkuPCuVPZVVIWtDPYPy+L204YxOSb/bMn10w/\niFNHd68aX1HhP3N+1B/n8+P24qDPq9z5Av93LD3FX78j7eTcf8ZwJoUc9NtZXMZBv/fnZ//9WrFs\nfc1J/SmjulFSVsGFk3vx0PvfceebdTtbNaZnLg+fM6pOdRqCutY0MCXyEk8HzP4Pxe709PDubXli\n1pgay2/bXUrr9JSqZKS8wmPVxh30bN+qTkf/Ze94nse7KzbRukVqjYlWpWgTeYAFyzfy3KdrOXFE\nVwbmZzPn2S+CLq4Ot9GvaXoDumTx7IXjaq0TanHhZi5+5GN6tm/lH42PcM1JXfy4vZi2Lfd0X4rk\n9a838MSHqzl5ZDfG9IrcVSxam3YUk5OZVus68tF3P/Hp6i3MGNyFrJapPPvpWq6at4QJ+7fn9hNr\nvpVubYl8oLLyCk6/bzHL1m/nthMG06N9JiOvmx9U5n8OP4DzJ/WqcZqBy/nkUV2ZPbVvtR2kkrIK\nXvpiHYtWbaZn+0xOHd09aAc28DMWXTmFDm1q75pYk9vnf8Odb67grPE9uKyGGxt8v6WIsXP9eJ00\nsiv7d2hFRloyM4fWvJNa+Z0m3PA667bt5tpj+pGbmcbseUtIT0niiiP7MM11zQtVVl7BUx+tYc6z\nX9K+dTr/OHMEeW0zgnYczxlfwIWTe5OV4Z+R3bB9N+0y06tidsXTnwXduSx0Oe8uLef9VZsZkJfF\n8h93cOJd71FW4THvgrHMnvc5S74PPgI+rFtbnvxV5N/9L9ZuDUrks1umcstxAxnTsx0pSVa1Lr24\nZB2zHvTzmBEFOTx27qiq9j7/q/Wc9cCea35mT+3D2eN7RJxmbQKXHcDXfzg84o75L+9+jwXLN1X9\nn5WRyugeufz15CFh18dPV29hwYqNzBzShfsWFvLXN/zkPC87A8/zWOtugHDLcQOZMaRL2GkuXLGR\nk/7+PuCf7Xj3t1O4df4y7ltQGFQu0jVwjU2JfANTIi/x9P7KTfzy7vdJTjJeunQC3dtlxnuWpBHU\nJZEPtXVXKZNuep2tRaXccdIQjuhf/WhlqEcXfcdvn/4ciO6MQSSRusXsS6I9cxN4sT9Et8MVGN/A\nNjIwPzuqh/QF1rnrlKEcdlDHWuuE+qBwMw+8+y3TB3bmkL771V4hCuUVXlQ3XVi0ajNL123j2CFd\naBXmqH1j8TwPz6NquW7YvpuFyzdx8IEdqhL4SK54+vOgMxy1LecN23eTlpxEdsu0am1kTM9c/nT8\nIParYecpNJFfcd2RNcY23E5rRYXH0Xe8w5Lvt3HlkQdy7oSeNc5zNN7+5kc+W7OVE4bnkxuh+yD4\nSf85D3xASrJx92nDgnaKonHfglWs3lzErIk9aN86nV0l5WzaUULX3JoTcM/z+Oi7LXTNaUn71ul4\nnsef5y9n6fptnDehJ706tAp7pigW1EdepBkZ2SOXhb+dTHpKMlkta++TL4mvrkc8s1qmsvC3U9ha\nVErHrOjqzhzaheyWaeRkptU7iYfqt6ndF9Ul6airwPgGXtx5/c/71/mz6psID+uew7CAC/4bQrR3\nThtRkMOIgoaddjTMjMCm3aF1C44ZnNco0wq8+ULo+vvfPzugxiQ+nNpiGy6pTkoynr1gHD/uKK7z\n9CIZ37s943u3r7VcXnYGz188rt6/JYHXbwFkpqdElYCbWVA3ODPjkkN612semhM92VWkEXRo00JJ\nfDN348wBpKckMb53OybtX/vGL1RGWnLUSTz4F/we3q9jXJKkfVVmeviuBdE6emBnHjlnFC9cPJ4D\nO0b3NO9LXWLSL68No3vm7tX0JTqBT0avj9tOGERWRiozBucxuGv46y2Cp9cw24akJGuwJL6udECg\n6dAReRGRevjFsHymDewcsR+pJL7WLVL5758dwMPvf8eFk2vu2x6OmdU5Gb/0kP2ZMbgLnbNbKFmK\nkfMn9eLB975lZ0k510w/qM71pw/KY9qAzlGf6cnPacm0gZ15/rO1NV5zIBIN9ZEXERGRfdrmnSWs\n3VK0V93W6mpHcVlMryOQxqM+8iIiIiJxkpOZFtPb7kL9r4EQCaQ+8iIiIiIiCUiJvIiIiIhIAlIi\nLyIiIiKSgJTIi4iIiIgkICXyIiIiIiIJSIm8iIiIiEgCUiIvIiIiIpKAlMiLiIiIiCQgJfIiIiIi\nIglIibyIiIiISAJSIi8iIiIikoCUyIuIiIiIJCAl8iIiIiIiCUiJvIiIiIhIAlIiLyIiIiKSgJTI\ni4iIiIgkIPM8L97z0ODMbFNGRkZOnz594j0rIiIiItKMffXVVxQVFW32PC831tNuron8KqANUBjj\nSR/o3r+O8XQTneJWP4pb/Shu9aO41Y/iVj+KW/0obvWzt3HrDmzzPK+gYWYnes0ykY8XM/sQwPO8\nofGel0SiuNWP4lY/ilv9KG71o7jVj+JWP4pb/SRy3NRHXkREREQkASmRFxERERFJQErkRUREREQS\nkBJ5EREREZEEpEReRERERCQB6a41IiIiIiIJSEfkRUREREQSkBJ5EREREZEEpEReRERERCQBKZEX\nEREREUlASuRFRERERBKQEnkRERERkQSkRF5EREREJAEpkW8AZtbFzO41s7VmVmxmhWZ2q5m1jfe8\nNRQzyzWzs83sX2a23MyKzGyrmb1jZmeZWdi2ZGZjzOwFM9tsZrvM7DMzu9TMkmuY1lFm9ob7/B1m\n9r6ZnVbL/J1mZotc+a2u/lF7+70bi5mdYmaee50doUyjx8HMkt3y+Mwt081ueY3Z2+/YUMxsvJk9\nZWY/uPXrBzN72cyODFNW7Q0ws6kuRmvccl1pZk+Y2egI5feJuJnZTDO73czeNrNtbv17sJY6TTI2\nsVx36xI3M+ttZpeb2WtmttrMSsxsvZk9Y2YH1zKdRo+BmWWY2dVmttTMdpvZBjN73Mz6RB+R6NSn\nvYXUv8f2bCd6RSgTkxiYWY75eU2h+b/Da83Pe7pE+32iVc/11Fz7ecPFoMjMVrnvtX+EOs2jvXme\np9devICewHrAA+YBc4HX3P9fA7nxnscG+p6z3HdaCzwE/BG4F9jihj+Je8BYQJ3pQBmwA7gHuNHF\nxAOeiDCdC934jcAdwJ+A1W7YTRHq3OTGr3bl7wA2uWEXxjt2YeY338Vtu5vHs+MRB8CAJwLa6o1u\nOe1wy216E4jVbDd/PwL3AdcBdwGLgRvU3sLO3/UB3+lu95v0JFACVAAn76txAz5x09sOfOX+frCG\n8k0yNrFed+sSN+BRN/4L4G/424qn3Xx5wMXxigGQDrzj6ix268rDQCmwExgZz/YWUndaQF0P6BWv\nGAC5wFJXZz7+b8o89/96oEec19MWwHMBcfiLa3cPACuBo5pze2uwwO+rL+Alt5AuChl+ixt+Z7zn\nsYG+52T3w5IUMrwj8J37rj8PGN4G2AAUA8MChrcAFrryJ4R8Vndgt1uZugcMbwssd3VGh9QZ44Yv\nB9qGfNYm93nd9+a7N3AcDXgVWOF+CKol8rGKA3Ciq7MAaBEwfLhbbhuA1nGM1S/c/L0Sbj6AVLW3\najHpCJQD64AOIeMOdvO+cl+Nm4tBb7ceTqLmhLTJxoYYr7t1jNvpwOAwwyfi70wWA53iEQPgClfn\nCQK2Zfg7bJU7H0m1xaMx4hZSrz3+Ovwo8AaRE/mYxAB/h8wDbgkZfrEb/mK81lNX/g5X5rpwy4+A\nbUVzbG8NFvh98QX0cAtjVZiG3xp/T20nkBnveW3kOFzp4nB7wLAz3bAHwpSf7Ma9GTL8Gjf86jB1\nwn4e8A83/IwwdSJ+XhxjdQn+UdEJwBzCJ/IxiQPwlht+cJg6ET8vRnFKwj+SshNoH0V5tTd/Hka6\neXgmwvhtwHbFzYPaE9ImG5t4rru1xa2Wui8TctAnVjHATwq/dcMLwtSJ+HmxjhvwL/xEPpeaE/lG\njwGQCezCz2dCE9Uk/PzHo4GPykcbN/xeEeXAIkJ6BdTwmc2qvamP/N6Z7N5f9jyvInCE53nb8ffc\nWgKjYj1jMVbq3ssChlXG5sUw5d/C/2EYY2bpUdb5T0iZvakTF65P3FzgNs/z3qqhaKPHwcV9DP5y\neLsO04mVMUAB8ALwk/l9vi83s0ssfD9vtTffN/hHPUeYWbvAEWY2Af8Aw6sBgxW3yJpkbBJg3a1J\nuG0FxCYGPYGuwDLP81ZFWSfmzOx04Bhglud5m2ooF6sYjAYygAUur6ni8p6X3b81Xv/QiE7E36F4\nAGhjZieb2RVmdm6k6wpoZu1NifzeOcC9L4sw/hv3HvZCi+bAzFKAU92/gStFxNh4nleGvxefgn9W\nI5o6P+Afne1iZi3dtDOBPGCHGx+qycTfxemf+N2QrqyleCzi0AtIxu9mEbpRjVQnloa79/XAR8C/\n8XeCbgUWmtmbZtY+oLzaG+B53mbgcmA/4Eszu8vM/mhmj+NvcF8BzguoorhF1lRj09TX3bDMrBsw\nBT8ZeitgeKxi0OS31y5Gt+EffZ5XS/FYxaCpx61yW5GF32X1n/hdbP4GLDOzOyzgwvTm2N6UyO+d\nLPe+NcL4yuHZMZiXeJkL9ANe8DzvpYDh9YlNtHWyQt4TIf6/AwYDp3ueV1RL2VjEoanHroN7n4V/\nNOgQ/KPJ/fCvS5mA3++wktqb43nercAM/CTzHOC3+NcbrAbu9zxvQ0BxxS2yphqbhIunO6L5EP7F\nf3M8z/spYHSsYtCk42b+nd8ewO/CcnEUVRQ3X+W24hrgA6A//rZiCn5ifz5wVUD5Zhc3JfKNy9y7\nF9e5aCRmdjHwG/wruE+pa3X3XpfY1DeecY2/mY3APwp/s+d57zbER7r3xoxDvNtu5REUA2Z6njff\n87wdnud9ARwLrAEmRuhmE86+1N7+B/8uNffjn97NBIbiX3PwkJndUJePc+/NPm710FRjE+91N4g7\nGvpPYCzwGP7dQuqjsWMQ77j9Gv+C4HNCdnTqK1YxiHfcKrcVPwDHep63xG0rXgNm4l+TdpmZpdXx\ncxMmbkrk907o0ZVQbULKNRtmdgH+KcAv8S/W2BxSpD6xibbOtijL17ZH3OgCutQsI/ioQE1iEYem\n3nYrN2QrPc/7NHCEO6NRefZnhHtXewPMbBL+Lc6e9TzvMs/zVnqet8vzvI/wd4C+B35jZpXdQRS3\nyJpqbJr6ulvFJfEP4p8Rehz/1qehiUusYtBk42ZmvYH/Be7zPO+FKKvFKgZNNm5O5bbixdCz3W7b\nsQr/CH3lfdubXXtTIr93lrr3SH2cerv3SH2kEpKZXYp/n9Yl+En8ujDFIsbGJbcF+Bc8rYyyTif8\nI4trPM/bBeB53k78xKSVGx+qKcS/Ff736QPsDni4hwf83pX5uxt2q/s/FnFYjn+lfw+3PKKpE0uV\nMdgSYXzlj3dGSPl9vb1VPszk9dAR7nsswv/dH+wGK26RNdXYNPV1F6iK0SPACfj3zj4pXP/iGMag\nKW+vD8LvdnRG4DbCbScmujLfuGHHuP9jFYOmHDeo47aiObY3JfJ7p3JjeZiFPNnUzFrjn0osAt6L\n9Yw1FjO7HP/hCZ/gJ/EbIhR9zb0fHmbcBPy7+Sz0PK84yjpHhJTZmzqxVIz/0Ihwr49dmXfc/5Xd\nbho9Di7uC/GXw/g6TCdW3sJPknpHOCXaz70Xune1N1/lHVTaRxhfObzEvStukTXJ2CTAuotbZ5/E\nPxL/D+AUz/PKa6gSixiswL/ZwP5mVhBlnVgpJPJ2ovJA2RPu/0KIaQzew89jxrq8porLew5z/1Y7\neBAj8917v9AR7tqMyoS5MGBU82pve3v/yn39xT7yQCj3na5y3+kDIKeWsm3wn8ZZl4epFJCgD5qp\nZzznEP4+8jGJA9E94KJNHOPzoJu/a0OGH4rf73ELkK32FjR/x7n5WwfkhYw7wsWtCPfE6X05bkT3\nQKgmGZt4rrtRxC0deN6VuZsoHngTqxgQ4wdC1SVuNdR7g8j3kY9JDNjzQKibQ4Y3ygOh6tje0vCT\n5grg0JBx17q6bzTn9tYogd+XXvgXk613C2Ue/mOBX3P/L8VtMBP9BZzmvlMZ/hH5OWFep4fUOYY9\njze/G7iBgMebE+bhDcBFbnxdHm9+sxsf+KjljW5YTB79Xs+YziFMIh+rOBD8yOmv3PJptMe81yM+\nHfBv0eXhH6G/yc1vGf79qH+h9lZt3pLwbzHp4ffDfgDXZx5/Q+cBl+yrcXPf9X73etFNe0XAsJvC\nlG9ysSHG625d4gbc58b/CFxN+G3FpHjEAH8nY4Grsxj/rmsP4/+e7ARGxrO9RfiMN4icyMckBvgP\nplrq6szHz3Pmuf/XAz3jvJ6Ow7+taZmLx03Am67eBmD/5tzeGizw+/ILyMf/8foB/5T1t/gXgtZ4\n1DqRXuxJOmt6vRGm3ljcQ33wjwR+jn91fnIN05rmVsLtrrEvBk6rZf5Oc+V2unpvAkfFO25RxrRa\nIh+rOODfovDXbrkUueX0AjAm3vFx85eDf3ZrlVu3NgHPAKMilN/n2xuQClyKf0p8m9vIbMC/F/9h\n+3LcovgdK0yU2MRy3a1L3NiTeNb0mhOvGOD3lb4a/yBBMf4OxxNA36bQ3sJ8RmU8qyXysYwB/m/x\nbfj5TQl+vnMv0KUpxA3oi39XpA1u/lbjn0mIOH/Npb2Zm5CIiIiIiCQQXewqIiIiIpKAlMiLiIiI\niCQgJfIiIiIiIglIibyIiIiISAJSIi8iIiIikoCUyIuIiIiIJCAl8iIiIiIiCUiJvIiIiIhIAlIi\nLyIiIiKSgJTIi4iIiIgkICXyIiIiIiIJSIm8iIiIiEgCUiIvIiIiIpKAlMiLiIiIiCQgJfIiIiIi\nIglIibyIiIiISAJSIi8iIiIikoD+H89yRsE8v3A2AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1a0fa271d0>"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 250,
       "width": 377
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(losses['train'], label='Training loss')\n",
    "plt.legend()\n",
    "_ = plt.ylim()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 显示测试Loss\n",
    "迭代次数再增加一些，下降的趋势会明显一些"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAvIAAAH0CAYAAABfKsnMAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAWJQAAFiUBSVIk8AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzs3Xd4FVX+P/D3JPQmTQQRBRcLFkRE\npKgI6Pqz97WXVdeyuva1F9QVUeFrAUUUFTuCglSV3juhhQ4hIYUkJCG93zu/P1JIbm6ZmTsz58zM\n+/U8PoZbZj53ypnPnDlFUVUVRERERETkLDGiAyAiIiIiIv2YyBMRERERORATeSIiIiIiB2IiT0RE\nRETkQEzkiYiIiIgciIk8EREREZEDMZEnIiIiInIgJvJERERERA7ERJ6IiIiIyIGYyBMRERERORAT\neSIiIiIiB2IiT0RERETkQEzkiYiIiIgciIk8EREREZEDMZEnIiIiInIgJvJERERERA7USHQAdlIU\n5QCANgASBYdCRERERO7WHUC+qqo9rFqBpxJ5AG2aN2/evlevXu1FB0JERERE7rVz506UlJRYug6v\nJfKJvXr1ar9x40bRcRARERGRi5133nmIi4tLtHIdbCNPRERERORATOSJiIiIiByIiTwRERERkQMx\nkSciIiIiciAm8kREREREDsREnoiIiIjIgZjIExERERE5kNfGkSciIiIX8Pv9yMnJQUFBAcrKyqCq\nquiQyMUURUHTpk3RunVrtG/fHjExctSFM5EnIiIiR/H7/UhOTkZxcbHoUMgjVFVFaWkpSktLUVRU\nhG7dukmRzDORJyIiIkfJyclBcXExGjVqhM6dO6Nly5ZSJFXkXn6/H0VFRUhPT0dxcTFycnLQsWNH\n0WGxjTwRERE5S0FBAQCgc+fOaN26NZN4slxMTAxat26Nzp07Azh6DIrGI5+IiIgcpaysDADQsmVL\nwZGQ19QcczXHoGhM5ImIiMhRajq2siae7KYoCgBI07maZwARERERkQY1ibwsmMgTERERETkQE3kP\nkeUxEBERERFFj4m8R/y6MQXnv7MQ78zZIToUIiIiolr9+vVDq1atRIfhSEzkPeK5qVuQVViGL5cf\nQHIOJ9AgIiJyMkVRdP03adIkS+MpLCyEoii4+uqrLV0P1ccJoTwoq7AM3dq3EB0GERERGfTGG280\neO2jjz5CXl4ennzySbRt27bee3369LErNLIRE3kiIiIihxkxYkSD1yZNmoS8vDw89dRT6N69u+0x\nkf3YtIaIiIjIQw4fPoznnnsOp512Gpo1a4Z27drh8ssvx5IlSxp8tqSkBKNHj0afPn3Qtm1btGzZ\nEj169MCNN96IZcuWAQDGjRuH1q1bAwDmzJlTr0nP6NGjDcfp8/nwySefoG/fvmjZsiVatWqFAQMG\n4Ouvvw76+YULF+KKK65A165d0bRpU3Tp0gWDBw/Ge++9V+9zaWlpePLJJ3HqqaeiRYsWaNeuHXr1\n6oUHHngAycnJhuMVgTXyRAaVV/qRkV/KZkpEROQYe/bswbBhw5CamoqhQ4fiqquuQn5+PmbOnInh\nw4fj+++/xx133FH7+VtvvRWzZs3Cueeei/vuuw9NmzZFamoqli1bhkWLFuHiiy9G//798dJLL+Hd\nd9/FKaecUu/7gwYNMhSn3+/HTTfdhBkzZqBHjx54+OGH4fP5MG3aNDzwwANYs2YNvvjii9rP//bb\nb7j55pvRoUMHXHvttejcuTOysrKwY8cOTJgwAS+88AIAID8/HxdccAHS0tLw97//Hddffz0qKiqQ\nlJSEX3/9FXfffTe6detmcOvaj4m8B3EQyuhV+Pz4+4dLkZhdjDeuOQP/HNxDdEhEREQR3XnnnUhP\nT8eMGTNw7bXX1r6enZ2NwYMH45FHHsGVV16Jtm3b4tChQ5g1axYuvvhiLFmypN5kSKqqIicnBwDQ\nv39/nHHGGXj33Xdx6qmnBm32o9dXX32FGTNmYNCgQViwYAGaN28OAHj77bcxaNAgfPnll7j66qtr\nf0NNUr9mzRr07Nmz3rKysrJq/54zZw5SUlLw6quv4u233673udLSUlRWVkYdu52YyBMZMH1TKhKz\nq0b/eXPWDibyREQS6f7iHNEhaJY46irb1rVy5Ups2LAB9913X70kHgA6dOiA1157DXfddRdmzpyJ\ne+65p/a9pk2bNpjRVFEUdOjQwbJYa5rPfPDBB7VJPAC0adMG77zzDq6//npMnDix3u9QFAXNmjVr\nsKyOHTs2eK3uMmsE+67smMh7kFyTCztTXnGF6BCIiIh0Wb16NYCqNvLBas1TU1MBADt37gQAdOnS\nBUOHDsX8+fPRr18/3HDDDbjooovQv39/y5PeTZs2oVmzZhg4cGCD94YNG1b7mRp33nkn5s2bhz59\n+uDWW2/F0KFDMXjwYHTp0qXedy+77DIce+yxeO2117Bq1SpcccUVGDx4MHr37o2YGOd1HWUiT0RE\nROQB2dnZAKqal8yZE/qpRWFhYe3fM2fOxMiRI/HLL7/g1VdfBQC0aNECt912Gz744AO0b9/e9DhL\nS0tRVlaG7t27N3gSAACtW7dGy5YtkZubW/vaPffcg1atWuGjjz7ChAkT8NlnnwEABgwYgFGjRmHI\nkCEAqmrn165dixEjRmD27Nm12+G4447DE088gRdeeAGxsbGm/yarMJEnIiIiV7GzuYqTHHPMMQCq\n2p/ff//9mr7TqlUrjBw5EiNHjkRSUhKWLl2Kr776Cl9//TXS0tLwxx9/mB5ns2bN0LRpU2RkZAR9\nv7CwEEVFRejatWu912+88UbceOONKCgowJo1azBz5kxMmDABV155JbZt24aTTz4ZANCjRw98++23\n8Pv9iI+Px8KFCzFu3Di88soriI2Nre0Y6wTOe4ZAUWNnVyIiIu8ZMGAAAGD58uWGvn/SSSfhnnvu\nwcKFC9G1a1fMmzcPJSUlAFBbi+3z+UyJtU+fPigpKcHatWsbvLdo0SIAQN++fYN+t3Xr1rjssssw\nduxYPP300yguLsb8+fMbfC4mJga9e/fG008/jdmzZwMAfv/9d1PitwsTeSIDVN4OERGRwwwZMgR9\n+/bFDz/8gJ9//jnoZzZt2oQjR44AqBpvPS4ursFnCgoKUFRUhCZNmtQm8M2bN0fz5s1x8OBBU2Kt\neWLw/PPPo6ysrN66a5r4PPDAA7Wvz58/v97natTU6rdoUTVU9ObNm5GSkhLxc07BpjUexM6uRERE\n3qMoCqZOnYrhw4fjjjvuwJgxY3D++eejTZs2SE5OxqZNm7Br1y5s27YN7dq1Q0JCAi666CKcffbZ\n6NOnD7p27Yrc3FzMmjULubm5ePnll9GkSZPa5Q8fPhyzZ8/GTTfdhLPPPhuNGjXCpZdeWvskQI8H\nH3wQs2bNwuzZs3HWWWfh2muvrR1HPjk5Gffffz+uu+662s8/+uijOHLkCIYMGYLu3bsjNjYWa9eu\nxfLly3HqqafihhtuAADMnj0bb7zxBi688EKcdtpp6NixI5KSkjBjxgzExsbiueeei35D24iJPBER\nEZFHnHzyydi0aRM+/vhjTJ8+Hd999x1UVUWXLl1w5pln4r///W/tOOynn346Xn/9dSxZsgQLFixA\ndnY2OnTogF69euGjjz7CzTffXG/Zn3/+OZ566iksWbIEv//+O/x+P5o1a2YokY+JicH06dMxbtw4\nfPvttxg/fjwURcGZZ56J119/vV5tPAC88cYbmDVrFuLi4jBv3jzExsbixBNPxIgRI/Cf//wHrVq1\nAgBce+21OHz4MJYvX45p06ahsLAQXbp0wTXXXINnn30W/fr1M7hlxVBU1TtNBBRF2di3b9++Gzdu\nFB2K7eqOqTv934Nw7ontBEbjfF8uS8A7c3fW/psdq4iI7FMzPGKvXr0ER0JepPX4O++88xAXFxen\nqup5VsXCNvIe5J1bNyIiIiL3YiJPRERERORATOSJDOCoNURERCQaE3kP4qg1RERERM7HRJ6IiIiI\nyIGYyHsQG4VET+FzDSIiIhKMiTwRERERkQayDdvORJ7IAHZ2JSISR1Gqnor6/X7BkZDX1CTyNceg\naEzkiYjCOJhdjD0ZBaLDIKI6mjZtCgAoKioSHAl5Tc0xV3MMisZE3oPkuIckkt/OQ/kYMnox/v7h\nMizdc1h0OERUrXXr1gCA9PR0FBQUwO/3S9fkgdxDVVX4/X4UFBQgPT0dwNFjULRGogMgIpLV079s\nRk1ucO/X65A46iqxARERAKB9+/YoKipCcXExUlJSRIdDHtOiRQu0b99edBgAmMh7EussiLQ5Ulwu\nOgQiCiImJgbdunVDTk4OCgoKUFZWxhp5spSiKGjatClat26N9u3bIyZGjkYtTOSJDJB1+MmRc3di\nxuZUvPD/TseNfU8QHQ4RkWViYmLQsWNHdOzYUXQoRMLIcTtB5DAyjlpzMLsYXyxLQEZ+GZ6ZskV0\nOERERGQxJvIeJGddMkUrs6BUdAhERERkIybyREQhsMktERHJjIm8BzE3ISIiInI+UxJ5RVFuVhRl\nrKIoyxVFyVcURVUU5YcolneRoii/KYpySFGUsur/z1MU5Uoz4iUiIiIicjqzRq15FcA5AAoBpAA4\n3eiCFEV5FcDbALIAzAZwCEBHAOcCuATA3ChjJSLSRJIZuImIiIIyK5F/GlUJ/D4AQwAsNrIQRVFu\nQVUSvwDAjaqqFgS83zjKOImIiIiIXMGURF5V1drEXTFYhaUoSgyA9wAUA7gjMImvXk+F0RjpKFYy\nEhERETmfTBNCDQLQA8CvAI4oinIVgLMAlAJYp6rqapHBuQk7uxJpw1FriIhIZjIl8udX/z8DQByA\ns+u+qSjKMgA3q6p6ONKCFEXZGOItw233iYiIiIhkItPwk52q//8IgOYALgXQGlW18n8BuBjAVDGh\nEcmPlcdERETeIlONfGz1/xVU1bzXzDG/XVGUGwDsATBEUZSBkZrZqKp6XrDXq2vq+5oVMHkXm1wQ\nuUd8ah6embIZJ3VoifF39kWjWJnquIiIQpOptDpS/f+EOkk8AEBV1RJU1coDQH9boxLA51dRWuET\nHUatvRkF+H8fLcOdE9egpFyeuKg+dmI2H4ef9IY7J67FnoxCzN+Rge/XJIkOh4hIM5kS+d3V/88N\n8X5Not/chliEyS4sw8XvL0b/dxZgc3KoTWGvh77fiF3pBVi5LxtjF+0VHY4UmOARuUdeydEB0TYk\nHQnzSSIiuciUyC8DUAngFEVRmgR5/6zq/yfaFpEAb8/egdTcEuSXVuLuiWtFhwMAOJBVVPv36oRs\ngZEQ2YtNqIiISGa2J/KKojRWFOV0RVH+Vvd1VVWzAPwC4BgArwd85zIAlwPIA/CnXbGKsCv96PD5\nBWWVAiMhp2HOSURE5C2mdHZVFOV6ANdX/7Nz9f8HKooyqfrvLFVVn6v+uyuAnQCSAHQPWNQzAC4A\n8IqiKBcDWAfgJAA3APAB+JeqqnK0NyEiIiIi6aiqaniCUqcxq0a+D4B7q/+7vPq1k+u8drOWhaiq\nmomqRP5DAN0APAFgGIA5AC5SVdWzw09W+vzsaGqxskof/ow/hMQ6TYlCkbHJhTeKLAKALcm5mLP1\nEMor/aJDIXKMxbszccNnK/H1igOiQyELvT4jHn3emo8p65NFh2ILU2rkVVUdAWCExs8mIkzOoapq\nDqpq5p8xITRXOFxQhmvGrkBReSW+f+AC9OnWVnRIrvR/8/dgwtIEtGgSizUvD0ebZo1Fh0TUwMHs\nYlz/2UqoKvDKlb3wr4tPFh0SkSP885v1AIBNB3Nxde8u6NSmmeCIyGwHsorw3eqqkaee/20r/nF+\nN8ERWU+mzq4UwoiZ25GeX4qC0krc85XYDrBurvWdsDQBAFBc7sOPaw4KjoZkIOOT2VF/7qx9IvTO\n3J1igyFyqNTcEtEhkAUOF5SJDsF2TOQdYFd6fu3f+aXsAGsHf4S2MzImeBK29nE8GZtQ+dmahoiI\nqjGRJyIiIiJyICbyRAbIWFMr4UMCIiIishATeZdauDMDH87fg6xC77UXI+PKKjkykuxUNqIiIqJq\nTORd6EBWER74dgM+XrgXr0zfJjocxKfm4Y9th1Dhc07jXhnbwFvtl/UH0XvEPPzruw1QZXzkQERE\nFIYXr11M5B1A72E5LS6l9u+/tmeYGoveCRaSc4pxzbgVePTHOHyz0jlj9zqxLIg25Bd+24aySj/m\n78jA+sQjpsRE5lPYiMpS3LpE5CRM5EkXvXe7H/y1uzYpHjl3lwURkRW8OIQXESDP6E+FZZUY/ddu\nfLZkHyod9DSTSCSvzOZalykTQpFcZKpN9skUjA6RygIZywoJQyIig8Yu3IsJy6rmtmjXoglu73+i\n4IiISEaskXeghMOFUX0/mtzai3e7wTj0/kQz7mZ5sbOrN9Qk8QAwbtE+gZEQkcyYyDvQsDFLEZ+a\nJzoMcjG336gQEckiq7AMfj8LXTKGibxDPTF5k+HvsrbVnXgZ8AZ2drUWty7Z6Yc1Sej/zgJc+cly\n+JjMR42j1pBjFJRWig5BE14UQzuYXYy84grRYQTFm70q3rskEHmDLM1EX/09Hn4V2JVegNlb00SH\nQw7ERJ50kaPoc74/4w/h4g8WY8C7C5GZXyo6HCIiT5Gx5ja7sFx0CORATOQ9KJryS76iz5ke+SEO\nAFBS4cPbc3aaskzeZJlPxm0abWfX6ZtScMeXa7B4V6ZJEbkLyzgi55LlSYudOPykC3FUi+jZ2Q45\nu5BjtpM9issr8fQvWwAAq/ZnI3HUVYIjIiKiaDCR96BobljddK9b4fPjPz9tQkpuMUbfck6995x4\nM+S8iMmIaG4ycyXtk0FERMYwkSdLyfyY69tVifhzezoA4P5v1guORi7y7jUia/HYJ3IuGfs+WI1t\n5MlSMp9Uq/dn1/6dlle/wymH+CNAzqcc0TwtkvH3kHU4NnloMlcyEenBRN6DJM6tbeW2zcDLEhEB\ngM+v4q6Ja9F/5AIs33tYdDhkofJKP8or/aLDIIGYyDtUuGTcykRdbyUGaz2IiOw1dUMyVuzLQlZh\nOe7+ap3ocKQk89NirRIOF2LQqIUYNGohDmQViQ6HBGEiT7q4oOxzLe4a88l4G8pmX9Zyw3m0J6NQ\ndAhkgN56r6d+2YyswnJkFZbjqShme3cTL1YeMpF3oUjHsQePcyKCO2ohKTInjrpF+ivKtqbk1f69\npc7f5C1M5B0qXDIuU9MamTGpISdikmYtFxVxFIYXa269wIvXdSbyRC5h5mWJ17gqbrskMHkhciee\n2t7FRN6hornp9OANq+W2p7nrsSaPEXfyYm0VEZGbMZF3Al57LRHNZg387j8+Xx1NKACA3OJyvP/n\nLny/JokJFxGRx7BWnYzgzK6SkT1/0ztihpXlUlpuCX7bmIKLTj0Wfbq1tXBNkRWV+6JexjtzdmLq\nxhQAwAntmmPoaZ10fd/MQ8dNF5TU3BI888tmtG7WGB/f1gctmzq72Itm1BrZyxcyB/dzZDJWlkgY\nEjkAa+SdQOd1O1JZ4JYkbdCoRRgzfw+u/3QlissrTV12xJF/TF1blZokHgAmrUy0YA3e9OyUzVh7\nIAcLdmbg44V7dX1XxlOFnV2JiILzYj8gJvKSCXoMBrlue/BYrWdDYk69f2/l0FtSJp0yWJNw9FiZ\ntz3d1nVX+PxYsjsTR4rKbV1vKF4vN4hqeDHh8wIZn7RYjYm8Q1nZ2XV3egG+WXkAWYVlxldisQSb\nZ7E7XFCGb1clYv9hTrTiJdFeEl6fEY/7vlmPqz5Zjkqf+GnUPXiNI3IM3luQEc5uLOpCQS+0Np7c\npRU+3PjZShSV+7B0z2FM+md/+1YeoMLnx4IdGejWvgXO6nqMsDgA4OlfNmPFviy0b9kEa18eLk3j\nhpraB9YuyenndckAgLS8Uqzcn40hpx4rOCJ3O1JUjrYtGvN8IM/hEe9drJF3Ahuzxm9WJtZ23Fyy\n+7B9Kw4aywE8+mMcrh67Akv3mB+LntrJFfuyAAA5ReXYdajA9FiMyMwvxRUfL8ffP1yG5Jxikw8T\nfZeF7MIyPP5THJ7/dQtKTOj460Y+v/gaeTf7cW0S+r2zANd/uhJ+vyy32kTa8YkZGcFEXjJ2VCSF\nW8d7f+6yPgCNRs49Gsu9X69DeSUTobpe+T0eu9ILsDezEM9O3SI0ljdn7cDsrYcwZUMKPluyT2gs\n4bCm1r1emR4Pn1/FlpQ8/GVzXwgi0Vi2eRcTeclovSN3yjlrZpzhJl3KLa7A27N34NPF+zxTG7di\nb1bt3+sO5IT5pPVmbkmr/XvKhmSBkYTnkNOGopQlSediMl9ecQXmbjuE/NIK0aGYzinXdZIL28g7\nVLiEP9LNgFMf34UL+90/diIpuxgA0OWYZrix7wkWxqFKkRAGDkMoQ0xERFa655t12JKciwt6tMcv\nDw8UHQ5ZJLOgFPsyCjHg5A6IieHVLRzWyJM+kp5PNUk8AExalajpOzLfz4iumYlm/bLfKO7JKMDo\nv3Zj56F80aEQWcKtQ/CVVviwJTkXALBW8FNIsk5RWSWGj1mKOyauxUc65/7wIibykhGdwDmGTdep\ngtIKxKc2bNIj42XSzJhcmgcAAG76bBXGLd6H68atdG3CQ0TOw+Koyk9rD6KgtGqSx0+YyEfEpjUS\nycwvxa50OUZEoaran6GjlyCrkO1t3aSgrOoCUe7zo9znR9NGsYIjIjIXOz6Sk5VVcuQzPVgjLwmf\nX8UNn62yZV12lvGWrcqEBUeqjf153UHPJvFuzQOyPd4JkjV+3sAnTc7k1nKXrMVEXhIbk44gNbfE\nlGUFdoJs8L4byniLf4MCoLD60Z5T8BoQWV6J+0a6IHMZqc0uq/Rhc3KuZ0bMIvmw/PcuJvKS8Lsi\nu5bD1pQ8PDl5Ew4XlFmyfO4qcirW+JlPVVXcOmENrv90Jf7761bR4RCRxzCRdyGF9+aYsTkNb8yM\nj2oZTsvXnRYv2Y83oZHpbZZyMKcYm6tHUvktLsWKkIiIQmIi70KRmtZEQ+QtQr3rq4ZA5m6zZnZH\nWWo1ZU3KJA2LyBIVPvlmnOY5SOQdTOTJmWy4UoVKlGVNoM0kyb0KERFpIEsFE9mPiTx5RoXPj8Ss\nItFhWIbluH5shkZERE7GRF4S4dIJrRXA6Xml0o2aIMt4xn6/iis/Xo5LRi/BuEXaJpiwsokSyUHE\nPl6TkI07J67Bd6sTDX3fC0+ERJKlzCK5qKqKN2bE4/Yv1mA353shiTCRd4lPF+/DgHcX4sbxq6Rq\nIGnZeMY6r7ULd2Vib2YhAGD0vD2RF6/YlzAFyxuMpBIS7XbXsOIYuO2LNVi5Lxuvz9hu2pCzWvHm\nlKiK3jJ2zrZD+HZ1ElYnZOO+b9ZJERM15MUSjom8A2g5uT/4azcAYHNyLlYnZJu27onLE0xblql0\nnq0FpfrGD7ez1jPYuhbvPoxV+7PsC8JErDHWLuFwoe7vsMKYIuE5GJneTbRi79Hy+FBeqbnBVONu\nC+7F3zisazhM5B1A78mdHzDpzTVjVxhe9//m7DT8XUDux9SyX+zu+HIt0i26YFCVSMeAxIevMLvT\nC1BS7t4p1GWcFZXHIUXi5v4+k9cnwydZs2GZMJH3gG2peaYtyysXFEUJfQMV6vWiMvNngp2xOTXk\ne4FxeGTXkEDjl+zH5R8tw9DRS1BW6d5kXjYS3ltIQcabLhLLi9dBJvIOoPfAlLlo8/tV5BSVG/y2\nzL8M6Pv2fMzemqb7ezLeHMn8JMVKRWWVWHcgR7ran7qd2EXmLu/9uQsAkJ5fit82hr7JJLIC83ai\nhpjIO4Cbyq5rP12Bfv+bjx/XJtm6XkN5qc6rRlmlH4//tMnq1YRejjmLqVqWzVdMVVVxxPANnjn8\nfhXXjluBf0xYjVd/j25WYDOt2p+F/iMX4LYvVqNSosmHrHgCZZa4pCN48Nv1+H6N/nLGqzexTvTX\nduOT/sm4l2WMieTHRF4SXrl4xKfmw68Cr0yXJ1Eywq4815k3cfqjfuDbDTj37fkYu1Db0KBmUlUV\nG5OOYPzS/dh/uGqegZ/XHax+z9p1a2nXeseXa5FVWI41CTn4ad1BaZ7gyDwCzvRNqViwMxOv/R6P\npGz3zh0Risz7xkyP/BDnqv0b1V6TpFwQzRtHfn1M5B2A56c1vHKxM8LOG8vd6QVYtCsTADBmfuSh\nQc22MekIbhq/qnbkJzvpPQajHb/ai00T4lPzRYcQNVlu3kQLdr78GW+8Vp7k5JWKTbMwkXeA4Nfe\n0FdkL16srRBuM5pZzoRblheKs3ydQ4OaSVWBx36Ks21dREbw2DEfN6m82IlZHybyLhSplq/C50dy\nTrEtsZiZiNp9bodan9mFjFmLc2vSn1tcbunESWWVodudB95kpRwx97wxMmQcr3HWYhJBori1DLdS\nel4pPvhrFxbvzhQdijBM5B0r9CkfKTm47Ys1uOj9xQ2miD9cUKZhrc4savTGLcvvDJtSBLxZUuG+\n4QCTc4pxwciFuOi9RVi657Dpy1eh6kqMn/lli+F18WmxPIrKKm2fVddObr0X0fu78ksrUOrCctHt\ngjWtCXWD/eTkTfh08X7885v1OJRXIsmV215M5D3s9Rnb6/37903eGk4u3EXBie3n7/7KmmnDRXp5\n+jaUVfrhV4F7vxb/+9Yl5ogOQZobAqcki2sP1J/pOqeoHAPeXYgL31uEP7YdavB5s9rn/r4pFf/8\nZp0pMzTbvc9VVUVqbomjn05sTDqCC95ZiAHvLkRakJs2SU6jeqLZ2jL+HqP0HHdrDxwtkxfs9Gat\nPBN5FzKahDoxebVSuO1h5vXN8EXawpJblotCZn7kp0TRCnfRcHAeE5TLfo4m362uGoKyZnSi56Zu\nQUFpJVQVePRHa/pH5BVX4KlfNmPx7sO448u1lqwjnGgT/2enbsHgUYvw7BTjT6DsEmoY1Hu+WouS\nCh9yiyvwyvRtNkdFonixjGskOgAyypzDNa+4Al+tPIAT2jU3ZXmyKa/0w28gG1Og2DfEZJj1hL0e\nS1piOSn5dVKsNWSJWZYnA1p9vyapwVNIqxzKN7fZjt59Hu0xMi2u6unstE2pGH3LOYiJkWNnB/tZ\nnyzaB78KPHf5afVeLyo/2qRVJKUHAAAgAElEQVTmQJYzhqiUYyuLx1Fr9GEiL0BucTkmr0/GGV3a\n4OJTjwVg/KJYUFoRVT436s+d+HldMgBg2OmdolhSCILPx0GjFqHC58dt53cTG4hBkuRswqTnl4oO\nQSoyXd9kuaHQyq4k3m1kOuZCGbd4X20ivzk5F5NWHhAckTNU+Pw4XFCG49vKVZFnuEmX0wolkzCR\nF+CtWTswrbo9+tL/XoKTOrQ0cPwp2JNRgOs/XRlVLDVJPIDasbzDWZ2QjddnxOPNa8+0/a7ZyCma\nVVjVNGPCsoSGy4uwwPUh2kOrcMbFzSnC7Ye8EuuHpvRm0U9WclM+oarOKu+0XhNdtIsA6N9HlT4/\nLv9wGRKyijDimjNw3+Ae1gRmksD99cFfuzB3G+cQANhG3jbfrU7EJR8sxg9rkmqTeACYvD459Jci\nePynOBSX+1Bcbm+v/O9WJ2lK+p1MhYr1iUdsWZeTLpKijV2419QOeJGWZPW+MbJ8Hi/uVRiivTf3\neRWj575TknY745y+KRUJ1U2ORszaYeOaI4tUSXggqwifLt7fsMmUR08UJvI28PlVvD5jOxKzi/Hq\n7/H13qspl4wcf/syC4O+bkdtUNxBe5JcUUbO3RXyPdFFRaUv9LjnWs3emoYbP1uJaXEpJkRknzHz\n92CJ2cNQhhu9KMh7Th7Jw8mxu92H8/fg7BF/4elfNke9rGj2sleOETPKcZ9fxcPfb8DwMUuwOTnX\nhCXaJ7dY3ER8RtSd1TrcfB6ir88iMJG3QbjOlsZHihFb2MoyzroIKsy9WdKzrLdm7UDvN+fhy2UJ\nUY0y9PhPmxB3MBfPTNmCChNuDOqy+sict13s49Th/7fUtgnVgvFInuU5Hy/cC1WtqinNCOgbInKf\nu+Fws+pq9fO6g/hrewb2Hy7CPyasjnp53r2qRnb12BUN5r5pQPXm2HtM5B1AxhoSjz7BEiqnqBxf\nrzyA4nIf3pm707TlBkvkZd6/ZvbNMHJuJRwuwrNT5R+Wz2rylUruYXdzSaeQ7ZirWwtfHmaGaBnJ\nXMaHUtNhXcKUSCgm8i7khYPcjPtuWe7dtRaoocZLlo3V1wcZrj/rDoiZGGrnoQKUOSxhqKGqKt7/\ncxce/yku7KNx57H+iBSZdMlYkaSXVb9Apk2j9ym5TLFTdDhqjWgaTqbgNZBi0xmtazfaBCe/VN72\neywAw9O7eQ7qbKYSY3JW46TduTHJuX1TZm89hM+W7AcApOeV4tdHBwmOyCxOOoIiY/lGsgt5iDrx\nMYMJWCNvAy0FY7jDT8oaEY0njJFa74z8UgwYuTD86kXfyJi4ehl3r52e09lM5fs1SaZMew+4LQWT\n25yth2r/3iDxDYkZp7boc1r0+kVYm5AtOoSgCssq8ePapHo34T6/+B3k0ZzXlZjIC6YG/F/PN906\n+9mImdulbiOqKNFdKKdsMD7kKFW548u1yCkqN2VZUt4okzAyHg3s7Fol3Ha4c+Ja+wLRYcy83Xhl\nejxuGr8K6XmleG7qFvR5ax5+2+isEcNkEjLz8WhZzkTeBtG2xZYxYbeyaY2W2TxNaSMvYPK4Vfuz\n8PyvW+u9JuHulTKmQMv3mjwMpQB2b2aRlzlZ+qSYzwEniwdUWlzLbbRM/GZlYu3f//11C37dmIKC\n0krTO8w7ocw2i1tLEqOYyNvAlU1rTJZZUIqD2VVtpUP9XLM3Q0mF/bX+X69ING1ZHjgsLCd6GwZb\nvd+v4rEf4zBszBJsTBLTqdZuezMKcO24FXjouw3CR//YGGJG59CMHUQ+v4qZW9Iwe2tag6YWgdcD\nO5O0wF8j+hyRhRnbIXBYUSIzsLOrYEaT9KxCc5oVyGBfZiEu/b+lAIAbzu2q6TtmtJEvq5Bj9I9w\nh4BdF1Ezx5L3wo2nlX6LS8GcbVXtyW8aH/3Y1FYxczc/+N0GJGUXYyvyMHFFAv59SU/zFq5TWl4p\ntqbkovcJbS1dz+ytaXhyctXkT+Pv9FB1ajRYtBDCXGO89FiiDtbIC6aptl7Cg9OMkF6Zvg2FZZW4\nbtyK2temb0qNfsGkS05ROS58b5HoMIQRmRsczClucFHampInKBpxkrKPjlz0/p+7BUZSRW8HbCNq\nkvjAvwHUVmwYJzbjTc8rxRGT+rDIzL3NxeS1L7Mg9JserURijbwDyFjDqbVGPFzC/+Pag2jeOBZF\nBjq2iixAxy3ai8W7zWufHW4b2XEPN3LuTmTkl1m/IhkJPrVemrYNq/ZnY+zt54oNhOrRN6qI+Sep\n1e299dBb1q7an4W7v1qH2BgFfz11MXp0bGlRZPYyoyyW8FLuOKP+2I27BpwoOgypsEbeBnafvDIl\n/pFCmbjigD2BmMjMJF4G+zOLRIdgyMp95gxBKdqsLQ3bSFtFoqLBRRpuVL3Jr0w1u9FeP+6auBY+\nv4rySj+e/9XcJxsybSfZ2P3cPq+kAt+vScI2Dz5BlA0TeRuEK3ysKJbS8qzvUCOitc+hvJKj63fo\nSBHBtpsMbeSdaMqG4MO3bUnOxZLdmfBrSI69nhjIdNNf16wtaaJDoGp6D5G6p92hgGuR369i6oZk\nrDuQI+2xF4oV4U5Z79yhiN+evQOv/R6Paz9dgdzi+s2o0nJL8OC3G7BBd8fxyCRsaSycKYm8oig3\nK4oyVlGU5Yqi5CuKoiqK8oMJy727elmqoigPmhGrbKw40O2gefhJE0+6p3452o7UlOEnPZbE1dT6\narmAOvVGaUdaPq77dCXu+2a95v4WDssnomTej9Vd66zj4//5eZPUszubzezzTdZj+uqxK/DfX7fi\nHxNW2zpLsWXbI8rlPv/b1sgfktSv1ePgqyrwe52yVlVVDBq1CAt2ZuDmz1db0ldC0sNbGLNq5F8F\n8DiAPgBM6a2oKEo3AGMBFJqxPJHCFSJbqh9Lfbk8waZonEvWi1O0jN7s6Nkck9cdxDlvzsOLGi8c\nTr3JeWn6ttq/zR6n2UpOq520Q6Zj+m1En4Q79XzTa8eh/Nq/b/5c/4hMZp4mYxft0/V51gRrcyCr\nflNNToBoPbMS+acBnAqgDYBHo12YUjVMyzcAsgF8Hu3yZBefmoe/tmeIDkMXKws1b1zStDFrO784\nrWqEoMnrk7Ev09p7Y5H7r0LnGOSqal8SxVxdnxjHJE7W71i95UA05YYbD9Ng22PBzgzEp0bZvlvn\ndrZy28oyup2fBZ3tTEnkVVVdrKrqXtW8aqUnAAwD8E8AzuyJV0ekjeLU5jWFZZWYtPIAlu5xZudP\nWcobo23kjRbb+w9HTuSjfdRfVFbZoN0khWbXRTjweJLlHAhGlsTEDpHON737SZb9Kvsu3KYjkQ+6\nTSXZzoD+p3pmnl91l2X1sacAurb77vQww1W6hHSdXRVF6QVgFICPVVVdJjoeM7jxsbmiKPho/h6M\nmLUD9369Dnsz3H+yyMboUTXqj10RP/PojxuxYIexp0S5xRUYMHIh+o9cGPYmNauwTHNTHyuVhp0Y\nzLxzN9x1s24ZIXvyY5TeLSmyRn7/4SL8tPagbeuTuWmNTJcvo6HI8hsiHdJeunm1y03jV4kOwXJS\nJfKKojQC8D2AgwBejmI5G4P9B+B0s2I1U6STN7HOZCkyqTt05Pgl+4N+hsVSZHaX3YHHU0GQToWl\nFX48+N0GlOtsqlK7zLJKlFf6cfdX60J+5vUZ8Zhs8qgNerelJNf3eqxMOsxctNXJUYzgpObl6duw\nMUnL09KGcYpOHEWv341MGUc+0vtR7DjeBFQJ3ISFZZViArGRVIk8gNcBnAvgPlVVSyJ92CkinZrh\nzr/icvcfhF4iY1EbbrjSoigLwZKK0JN9zd2WHtWyzRD595m3x5hcOc/vm7w3DKadx+n3qxPtWxkJ\nkXLENamctKRJ5BVF6Y+qWvgxqqrq785eh6qq5wX7D0DkNgWSmblZzgtJ4M2HmWX/luRcE5cmTkFp\nBf7YdsjRbcXdXslz0fuLRYfQgJnbPKeoHKv2Z4WccCqa81ZvnHrP6xgJervK0uRF5Hlo5TZ4bcZ2\nXZ83WmNt1ayschwdVcSfLcF9vyYJf8YfMm15kfal269ZwUiRyNdpUrMHwGuCwzFdpLIn3HEnU0FR\nV2DnLDf2A4jWv3+Mw6M/xuHOiWst2T52bHOzx7eesiEZH87fg7xia8YIl7kQD9tGvu7fJu3Wskof\nLvu/pbjjy7V4/69dpi5b77KKyyuRWaBvOEkJ8nhpsHiNjlO2n53NY+w8vR75Ic7U5YW6uXTIbjZd\nI9EBVGuFquErAaA0xMH8paIoX6KqE+xTtkVmhiiOrmUOHRHGCaw+6ZfvzQIAbE/LR05ROTq0amrx\nGuW2al8Wnv+1qnOr3qROHOdeGmZuTkN29WQsE5Ym4KUregmLJT41P/KHAvy6IQXNm8TirgEnoVnj\nWAuiiswpCaCVuA2qmFOr7/6N6f5fKB9ZEvkyAF+FeK8vqtrNrwCwG0BUzW6c5o948e2Ig7GyaY0b\nWbF9FEWxaawvc9Sd9OzndeaMCOL3q0jNLcEHf+1Gj44tTVmmVbTuKrMq5cp9DTsqy9JURIsx8/cA\nAEorfHh82ClCYjBra61NyMb53dtL0VwoEjOPEdNnrDV1afaLVOPu1ERfliehkoRhO9sTeUVRGgP4\nG4AKVVX3A0B1x9YHQ3x+BKoS+W9VVZ1oV5xmilgwynIW6BAY8cGcYnzw1y5cclonnN+9vaXrzi+t\nwGaJ29EnZhWhu0lJ5fa0PJRX+tGkUcNWcJEK/Xu+XofSCh8+ue1cU2KR0dAxS9CsUSx2O3z4U1HX\n76pjyPryJ5oibvS8PcISebPc+sUaPD60J567/DRD3xfbRp4ABzyZsPEYSc4JPZKe1WGYPeeCG5jS\nRl5RlOsVRZmkKMokAC9Wvzyw5jVFUUbX+XhXADsBLDRj3W7gvDS+oU0Hc/Hp4v245fPV9YZ7MvsC\nVFrhw/AxS/HzuuiHLbSq9uPNWQ07cBld1eyth3DL56sMxbpsz2GsO5CDF6eJH6sdsKb9Z1J2sWOS\neAfer5vGwz+91rjF+2xbl6gnL6E6Vctq+qZUW9dnaY27jZv+6V82yxCGVOsWyawa+T4A7g147eTq\n/wAgCcBzJq3Lcbx2h7g9NQ8XnNwBgPm/ffqmVBw2qX21VbslWJOGGkaSuS0pedidUYCWTYydrkt2\nG+9n4eXkM5ra6rySCnyycK+5AekUdJQNQWWRl48jMzjlGjJ1g7nzQgQyezusO6B9VnUzjuFIlRlO\nGQt+Q9IRYes20qK0wudH41gpxnaxhCm/TFXVEaqqKmH+617ns4mBr2lctiOb1bhVuPKmbmFUanBC\noVCKy0OPSy4Lv7k/GQBQUSnmSu6My4o1erw0F1mFxm4ax8zbja/qTJgWiVk1qBOXJ+Dqscsx3+Cs\nvFSfUxJoMwX+Zj21yIt2Zepe377MQhwpcuYQvXpr2C2tkfdyYV1HsNzkF5MnHpSNe29RJBLNhFBO\nVPf3+GW+EloUWrCkLNpETVGcf5w4PHxdvludZPs6MwtK8b85OxGfmo9/fbehwfuJWUX4x+eixgrw\n0t73rsAySkuZden/LcXAUQtNe9JqFTsuZXZ2djXzeqJlUVd8vBzjFln7lDJUHJ/a2KxNBCbyNnBq\nT/RwwnU4qXln5b4szNlq3kQQgLnb0qq9ErSZaISVFZTqH1fdaUeV029EZJeRFz4ReuynOBQEzGTr\ntGPIq9ze2bW0wo9Rf0Ser9FJoy5RfTsP5WP0vD1hO8pqoXU+Di9hIk+GhG9aA5RX+nHnxLWmrzc+\nNc+0ZVl1gxVuuZW+4O+N/mu3JbEQ1diepn8sdzMkHC7EGzPjTV/u9rQ8HMqzevp3+VMDVVUxZ+sh\nTF53EGWVvurXxMQSzXCTss+AreVm6sFv1yMzvzSKdbi/tiP5SHSJfCQurDeNSJZx5F0tYtMalz12\nVhQFuSXWFMq/b04zbVlWne+hCpL0vFIsDNGGdMW+LIuiIav5/Sqyi8pxbGvjE35ZcfEpM7l/ilH/\nnLQeSdnmXrxnbE7Fk5M3o1GMgiX/vQQntGth6vJlE+74WL43C4/9VDVzZnG5D/df2MOmqCjQgp2Z\nAOIx8d5+tq9blizCjkR6b2ah9StxENbIS8BtN+EK4ISKLMsKnGD9AlQAr0zfZs0KLWTuJnLZgY6q\nJP6Gz1ai/8gFujq3BrIi6Z5YZwKuUJbtOYzKMKMsRVJcXgl/QFuyjPxSPPjtejz9y2aUVvhMT+IB\n4MnJVcPfVfpVvDlrR8TPFwU0KdLKaBlhZ63giJlHh7t9a3bkbaGXnt9i+bUsQix5JfqbKNY9T/x+\nFSv2ZmFPFEPaLthpXUfzpOwiQ7/RbbQ0w/ISJvI2cOOjnnCPAGMUJXg7cclY1d4y1G/flR784qDl\n+Hhj5vYGs6HadVwt2pWBGz9bie9WJ9qzQgdZuucwtqTkQVWBt6NIos55cx4mrTR+IxDMobzIj/gf\n+HYD3pm709Dyl+89jPP/twDDxiyplyi/+NtWLNiZiembUjF+yX5Dy9YjX0Ni8/nS0HGk6HjUL6q/\nk1PaBYuulArVRDFcWf+/OTuRV1x1DE3ZkIy7vlqLv3+4DIlZRabHF82+mr01DUM+WIJB7y40PJpW\nXU6tVjHSgsGNOVhdTORt4LUOOorivd9c1+bk3KhqdILZmHQEny62PikK5v5JGxB3MBevz9iObBMu\nIG5yxMR2vSOqa5btvsB+szLR0PfWJOSgqNyHxOxifFJnNIrFdeYtmLXFvKZw0Ri7KPSoFVd8vDzi\n9wvLKnHL56tw2YfLsC+z/rlt5jFgFlNLXwcV5d+vMTZaVE1T0BenHX1qOiLIxH6B7Nw0j/+0CQBQ\nVO7DeybUSDtot+oi+mZSBCbyEnDicRcuZgWKI+6ArYzx3q/XaV6XzAVPYNzpUXXkijIYCen9TUnZ\nkWv5jB6WIm+eDxwO/rscUAygoDR0s5ua43/MvN1Yn3gE+zIL8dD3G2vfzykqxy0mDOmptyz6Y9sh\nfLp4nyXNLKIpFwNrS80+5Y2GlpxTgsyC0GVXsN9cVmF+czeztkeObOPuS1K4vz5jOxJClEVuxs6u\ndohQ+khyDugSadQaqcePr2ZliFqaNThRdBd5PetRsXyv+zoA13RKdJtQh4UdTVHsKGlW78+u/btu\novDslNBT1Qdz3zfrMOKaM9G9Y0td36u7Gbel5OHRH6uOo9Rcq0ftcY+H69yABYp0DBWWVeLnddon\nFZqwdD8evOhkxMbUL/WsPFb1jnhjRdqxaFcGPlkodsz2lx3YFy1aTOTJEg7I423l5aZGV49djs5t\nmqFSR8eJDUlHcE/AUw03iE+NPASk0Qus20a/kkGk87ZuMyItluw+jH/nx2HukxfVe11PDjZxxdHO\nmT+tPYgeOm8K9Aq3DUrKfWjeJPboCxIfgpsO5ur6fN3f/ZnOCYXe/WMX2rVsgn/066bre06WX1qB\n+yc1nIjOdBIfY6KwaY0N3JjChRuJwhm18aqttVmqGukphqSlU4Pp2vUvIj41Hwt2ZmKJjqTnqcn6\najq97LpPV+KZKZuFnndCj14bfraZ5+eOQw1v5iLtOhmLh183puCct+bhji/XQFVVTNmQHHYCQC3N\nygLtTi/A879uwdxt5k4sqNdnBjptjwvTL8ON9nNISGGYyNsgYiHtwFvMSasSQ76nqvIn89d9ulJ0\nCPW4cfZfo5y0LWQ4d7ck52JaXCqmbtT+6N9sIZvWRPhepc+P+Tv0Dde3ygFzLtj5BC7wfPH7VcQd\nPGJ8eQGx55VUYMqG5AYzcj43dQvKK/1YtT8bs7YewvO/bg25zHf/2IkhHyzRHcvtX67BlA0p+PeP\nccgsKHX9k95Kv4q9GQVQVRWqqmJzci52pYuZyK3GxOUJGDxqEb5fnSg0jmi4/Yk4m9bIQHwuYCq/\nqkpf4G5NMW+GWDcLLADtKBCX7c1yTDIfrKb0oAXjpmuxNiFHyHrDibQbv12dpHvYzjssmDE6FKOH\n4boD+vZFNDXuiQHH2ztzd5ra4e/F37ZhdUI2jmvTFCteGIbGsQ3r/3ZHSDYnLI08p0GwbVC3U+e2\nlDycefwxkQM2IFh5o2Xf6x7EIMIya86Fhy4+GRf0aI8Hvq1qqjLjscERY4n2qU1GfinWJGRjeK/j\n0KppVWpYVunD/+ZUDU/72oztuHtg9+Dr1rGeYJUfqlo1qV7HVsYn1fMy1sjbwO13g4FUuLM5UTRU\nOLRpTQA78usHJq23fiUWevxnd3ZoDcfo0RvN2Ps17Chf9f6+muRHq0jnVXJOCW7/Yg0mB8wlEUyw\nick+X7ofxeXGJsVanVDV0TcjvwxxScZr+vWyYhz3UMJt/nIzJ2vTeCB9sSyhNokHgCcnb6q/GBMu\nF4HXnJvGr8KTkzfj+V+31L5W4dN2buk5AwPP1/S8UvR4aS76/W+BpomenHGltBcTeTKdqqrSN60h\nY+zYq3o6xcqIT3vcw64jUct6Vidk48Vp2wwNPTjqj12mTM4VqsIhVHEfTQXFg9817DgpolIs3LVM\n988zGH5Ruc/YF3VIOVLVZ2zutvSwnwvVxMbIvlZVFVd8vKz23+EmbouGz+HXlEiYyNsgcht5d/Gr\nHLUmGBnaU+sVuB+d0uTFi2TcM255Gmn1AzM951W6waFtw02KVT8W/cu2Yi/vC+g8KWPRI2NMZgo8\n7Dcm5eC1GQ0nyjLatKbSr+JIsb65EIxUlGQVlmPqBnF9iKzGRN4Gkc71P+LD3wE7TVXh5vISTqdI\nF+rAixbJ74DOR/9uvuiL/GlWbtdwyw43Qgs5X82uD1c2zwwzc7HV1TZ15zYAgNziCmRGMWGfFrO2\nRH/M1725N3LuHswx1gfpv2E6YzsdE3kJLNqVKToEUzmhs6tMCsusf2zqNE44fJKyi8xtPysBv+SP\noGV6ImTm5F7y/Cpj7NgtqoXrCbpctap9/M2frwr6nXAzxZotML6ySj9u/3JNg88Nfm8RDuXZP0mY\nQ7p4uRYTeRvIdPGxw0vTtiHNpTObGhVuHPkSg53Q7BB45HrrSA5Phb5aHpmaZwTz4m9bcc6b8zBl\nvf5H0KF+WnaheVPJ/7zuIM59e36D1+04JmVKVGQ8B+1oQmXtdTT4sren5aG0IvjNelaBecd2JFmF\nZfX+nZlfFvRzFT4Vb82KvgO5lZzYxFR2TOTJdAeyivDc1C2RP0gAqvoUOIXH7kkdZX8UQw7uzSjA\n5PXJKCirxPO/6X8EHeqwKDaxk95L07YhV2d72mjZ1cZfpvPKzFCYslkj3I1lXol554jllQ9S3pY6\nDxN5G8hUSNvlcEHwGgOvKiiVt9ZdRo44Z3TGKPNvyuT5ShqIfLrstGGNRQ0prHW1ZofHmzZxmMgT\n2WDMvN0hCzqZm141jE3eWO3mpiFWo70Ii7yIW3r+VC86PlXs7JrSc/ipoKoN+4dEqi0uq9T/tMmW\nZmBMqT2HiTyRDRaG6dDspHzQrlhlapNsFu01ZQJ+fJSrtPqwEDkO9JbkXGHrDkbkjb9TJq4zYt4O\nfaPHfRZhXH5RW8rFu4hCYCJvAyclamQdJ14ERdXHO+Gc0RujT+MANyISNdlr8W4cH3zkEMD6Y3Kv\nzqFhZX7CpoWs8auqdbGpAD5asLfB6+HK7Pk7MnSvR++QtaJZXS5Ieqg5DhN5IsGcVJax4D1qbry+\nMZWnxaVYFEn0ZL/HFFUrbuRwXxUwvrdsknOKsTEpx9Sk+Ke1B01blixkLutkqxSSvSLA7ZjI24A9\nsykcozM1upkTzplpcam6Pq910i/ZLtKkz0PfbRAdQkhpuSW4ZPQS3DR+NaYYmOkyVPJfUGZHZ375\nywQnMVLK5JswIg6LN/MxkbfBxqQjokMgCYQqv0oq5J0QKvC6LetjdyfQuuXENK1xLmtndtW/8FKL\nJwmL5ueOnLuztr/BC79tMyegcEw8sKqa1pi3vMBlm07QSaUoCpJzinHrhNV48NsNKDZxnpJpm0JX\nXmhN0Otua15OzMFE3gbPTOGY6uQOLHfdiU8Bgtuakqf7O1bfiCVEMV+AlnH9w0XvteNE1l8bKa5n\np2zB2gM5WLAzAx8vbNj2XxZOePLqBEzkiUgzu2pQMkLMXKhXpdYepjbQmuCJSJaiXeW2lDw8+sNG\nTDXQXENmCQ7rnBhJtEeWtTcp4aOzdl7Xhkt3aoqpAFiXmFP7byOdcmtYkWh77F7QFo1EB0DkGQ4s\nwAILcqfVoPR85Q/RIdTSsuV2pOWj0i/PzYdW6fml+CM+HX/Eax/Cr8KkmyzZjkjZ4qnLazXqWu1I\ny29wPEZ705JXXGFZRYKu3WjDAWn0sGLTGnMwkSciksSVnywXsl4R6Z1TRjoZu0hf0wSZkxMn5/FW\nblcrmr9mF5Vj6Jglpi83ksB9bNfEdRy5Rhw2rSEi7YJcE/7arm8iFc9iglePaWNqW5yoJGUXm77M\n71Ynmr5Ms8h8I+I0yTkllixXz/kq8+6UOTYnYSJPZBNH1lcElLS7MwowcXkCMvKrhszcl1mAh7/f\nKCAw55H7ouXIo9OxXp+xXch6Y6LczXY1zQnVrMXumwwnPsHQHLIEP46joJmDTWuISLM3Z+0AACzY\nmYHJDw3E1A3yTnJE2om4pkuQR3hOtM0f9CZeRtb29uwdmLUlreG6bb4VljnFjE/ND/le4M1WNE+U\nROXZLBv0YSJPRCGFKsfXJFSNimBX+0s3YO2TNbhVtYuR+Bn8gp0Z2J6Wh69WHBAdiuuUVvjQrHGs\n6DAaCHbuqqrK5l06SXxaE5Fof2w7FPZ9PwtczWTeVE6uAEvLtaYdshtpqpEXeKDe/sWakO+pqvNG\nzZJFQakdM+/qZ3fCnlVozrDGsmEiT2QTJw79NqK6KU0orDnRTuZt5cRjs0ZWYbnoEMgk+WESTrtP\nH1V15kgsIiI2c52qal3TmjcE9U2xGhN5IjKMTWvcwXnpChnh4Ps1IZz4BCDYPg72O7QcCnp+vaFj\nK8gKVFhX6bF0z2FrFveHUxkAACAASURBVCwYE3kiMqSqLaPzLnSiyJwUCOnsytsH29n95GX/YfNm\nxmXbaa2CZvJaXrJFpPkjrLymuPV6xUSeyCZuS1t8fplTU/m49BoihXfn7hQdgiNoq4UNf6D6dXaM\nyS+t0PV5mfBm03wzt6RhyvpkAMGPtScmb8KfOmaIJo5aQ2Qbtz3WrvSrbFrjEjEOPzgnLEsQHYIj\nBO7mvJIKHNO8sebvj/xjFxIOF+K5v5+m+Tv5JeYk8p8s3ItrzjnelGVp4dSSTeupLPKMf/63rbiy\ndxekHGnYUX3uNibxerFGnsgmezIKRYdgqgqfn6PW6MB7nvocfu/gSIGbXO9Qj1uSc1FQWok3Ztrf\naXD/4SJ8tGCvfSvkCau5KcryfVm6lz3w3YW4euwK3d+Lhlv3KBN5IjKM1zp3YFLtDYFt5Msqfbav\nk6wVbGtbXUzP2Rp+mOJgRAyJ6dbrFRN5IjKkanQBl5aM1IDettGRcOIf+zXIqYN1guQpXYv3IOQE\nTOSJyDC2kddO5pueSJ36Uo4UY8joxTZFQ1Zh503303rz4cWJ1Nw6PAMTeSIyRFVZexdoW0peyPfK\nffJurMCL/4bEnHr//u/UrUjO8d6F3+3kPSLFc+q20XKz9saMeHy2ZH/Ezzl1G4Ti1usVE3kiMoyd\nXev7LS4l5HsyTw8emMjfMXFtvX+vPZBtYzRklZgIOZ6qqtiaGvpmlJwpMIH9dnVS2M+XlPuQnFNs\nYURkJg4/SUSG5JdUhE1cyTkCa/HKK/1IzilGt/YtAPCGzS0iNbuYtCoRb87aYe46TV2avfJMGjrT\naS58bxGyi8rxxjVniA7FVG4txlgjT0SGXPQ+20y7xcTlDcdhf3HaVgGRkJUiNbswO4l3MlUFRjpw\nojEzOuhmF5UD4PHgFEzkiYhMInOH1nCmbmz4ZCU1yGQt5GyBSZ5Tj1e7bE/LFx2CbsESeTs6eXKE\nH3GYyBMREXmAiGSLCZ69ODJRGC69b2UiT0RE5AGBkzPZUSH/wKQN1q+ESAMOP0lERJ6RmF0MH3u5\nBmX25Fh2EVFXu+OQ85qnAA5O+gRVyIuYqVUvt7YkYyJPRGSSSMO6Oc3bs9nZLZiFuzJFh2DIYofG\nLYJTk77DBQ2Hua37W6zqFzFm3m5LlkuRMZEnIqKgJq1KFB2ClIIlS06QllcqOgSy2LoDOWHft+oG\nZX3iEWsWbCKH3ptFxESeiIhCiucEQa41ccUB0SFIy4kj1mjxloefsrl1lCYm8kREFNKtE1aLDkE6\nL0/fJjoE02xNyQUAZEs88zCZY3NyLp+yuRATeSIiCqmo3Cc6BLLQlpSqJy7//jFOcCRklZp66H2Z\nhULjIGswkSciIvK4tRHaVpPzubVpiVZu/fVM5ImIiDzqtd/jsSejQHQYZKGMfHZydjMm8kRERB72\nGJvVuNpD320E0HBCMK9x6wMJJvJEREQetpdtp10tq7AM87ane75pjVsxkSciIiJysYe+34itKRxK\n1o2YyBMRERG53Pdr3DXzNFVhIk9ERERE5EBM5ImIiIiIHIiJPBEREZHLxXh70BrXYiJPRERE5HIt\nmjQSHQJZgIk8ERGRhzWJZSrgBc2bxIoOgSzAs5eIiMjDmjRiKuAFzRszkXcjnr1EREQexrbT3sD9\n7E5M5ImIiDyM8316g5872pWYyBMREXkZEzwix2IiT0RERORyKu/YXImJPBERkYcxvfMGlTvalZjI\nExEReZjKDI/IsZjIExERERE5EBN5IiIiIpfjgxd3MiWRVxTlZkVRxiqKslxRlHxFUVRFUX7QuYwO\niqI8qCjKdEVR9imKUqIoSp6iKCsURXlAURTedBAREZmsqNwnOgQiMqiRSct5FcA5AAoBpAA43cAy\nbgEwHsAhAIsBHARwHIAbAUwEcIWiKLeobMxHRERERGRaIv80qhL4fQCGoCoR12sPgGsBzFFV1V/z\noqIoLwNYB+AmVCX1v0UdLREREZGHsB7UnUxprqKq6mJVVfdGU1uuquoiVVVn1U3iq19PB/B59T8v\niSJMIiIiIk9iGu9OTml3XlH9/0qhURARERE5UIXPH/lD5DhmNa2xjKIojQDcU/3PPzV+Z2OIt4y0\n3SciIiJytKzCctEhkAWcUCM/CsBZAOaqqvqX6GCIiIiIiGQgdY28oihPAHgWwC4Ad2v9nqqq54VY\n3kYAfc2JjoiIiIhIHGlr5BVFeQzAxwB2ABiqqmqO4JCIiIiIiKQhZSKvKMpTAMYBiEdVEp8uOCQi\nIiIiIqlIl8grivICgA8BbEZVEp8pOCQiIiIiIunYnsgritJYUZTTFUX5W5D3XkNV59aNAIarqppl\nd3xERERERE5gSmdXRVGuB3B99T87V/9/oKIok6r/zlJV9bnqv7sC2AkgCUD3Osu4F8BbAHwAlgN4\nQlGUwFUlqqo6KfBFIiIiIiKvMWvUmj4A7g147eTq/4CqpP05hNej+v+xAJ4K8ZmlACYZiI+IiIiI\nyFVMaVqjquoIVVWVMP91r/PZxMDXNC5DUVX1EjPiJSIiIiJyOuk6uxIRERERUWRM5ImIiIiIHIiJ\nPBERERGRAzGRJyIiIiJyICbyREREREQOxESeiIiIiMiBmMgTERERETkQE3kiIiIiIgdiIu8R7Vo0\nFh0CEREREZmIibxHtGzaSHQIRERERGQiJvJERERERA7ERJ6IiIiIyIGYyBMRERERORATeSIiIiIi\nB2Ii7xGKIjoCIiIiIjITE3lJ9e/RXnQIRERERCQxJvKSGnnD2aJDcIxu7ZuLDoGIiIjIdkzkJXR+\n93bo2akV/n7GcaJDkUaLJrFBXz/nhGMw4a5+NkdDREREJB4TeYlNuPs80SFI4/sH+gd9fXiv49Cy\nafAkn4iIiMjNmMhLSFWr/q+wh2qtcNtCAbcTEREReQ8TeY9werIbEyKRV1WOyENERETexESeHCFU\nrq5CtTUOIiIiIlkwkZcQU9OGQtXIA6yRJyIiIm9iIi8hVWUqHyhcss6+BERERORFTOTJEXz+0Dc3\nTOOJiIjIi5jIS4j18Q35wjylYIU8EREReRETeXKEcK2NnD4iDxEREZERTOQ9wum11v4wmTxHriEi\nIiIvYiIvIfZ1bcgfpo08txcRERF5ERN5CXk9L72013ENXgvXRt7r24uIiIi8iYk8SSNGARY8czH6\ndW/X4D2/P/T3OFwnEREReRETeZJGz06t0LNT66DvhaqRV1U2rSEiIiJvYiIvozqZ6bg7zoWiAG2a\nNYpqkQ7v6xq2sysRERGRFzGRl9zVvY/H8ueHYvVLw0WHYppzTjgm6Os1uXqwnN3NnV3792gvOgQi\nIiJyICbyEgrMS09o1wItm0ZXIy+TD2/to/s74WZ2dfLwkwueGYIpDw8UHQYRERE5EBN5CTm9hjmc\nIacei46tm4b9TLAx78M1rWnfskm0YQkx/s6+6NmplegwiIiIyKGYyJM0alL1YDm7L8yoNa2bNcYH\nN/fGgJPb44ELe4T83H+G9YwuwDrOPL5N1Mtw8f0aERER2YCJvEcoYaZ2fWL4KXhiWE/ccG5XzHhs\nsI1R1RduGMlObcLX4t/SrxsmPzQQQ0/rZHZYQb13U29b1kNEREQUinsaXruI3W2+HxjcA8e0aGzL\nusLcT9Q6LkjS3u+khmPLA/pqtc8LsQwj2pqwvdzchIqIiIisxxp5QkyYo6B9yyZoEmveYaJlGMzr\n+nRt+D0tdwAIf6NwXJtmmpZBRERE5ARM5CVkd01tTJjs9/i2zbDyxWFY+OwQdDnG2kS45mfHxigY\ndePZmr7j5PHxnTzaDhEREYnHRN4DQjVLqREukVdV4NjWTfG3Y+UcXSUwFbYisb9vUPeG6zUhBw8z\noiYRERFRREzkXap7hxZ4+tJTcX2f4/HRbX3CJrgaW62YQmsTGStyXKPJd4smseYGYpFzT2wrOgQi\nIiKyETu7SsiM2l5FUfDkpado+my4GnmzhV1Tnd/thY6g4UbpMcLJzYyIiIhIP9bIS8iM9C4wqbvw\nlI4hPxsTJgMUlVAHaz+uadQZC7LZYPc5XrjRICIiIrkxkfeI/15+Wsj37KyR10oJkpEHa6uu5Xs1\nmhtoItO6aSNcHzCKzlf39mNHVdKlY6smuPac40WHQURELsNE3sHO6KJ9dtHWzRqjcWzwJDcmXJV8\nHWak++HuGeqmxo2CxNq0Uf3DtXGsgnsGnqR5+T06tsSFPaueTPzrotAzwNb48NZz8Pvjg9Giaf0W\naMN7HWfopiAQa/UbevrSU0WHYLoBJ7fHiheG4UENxxwREZEeTOQlpKXt9F0DTsSvjw60IZqjzMk7\nw42Qc3QNgUl7ML89OggdW4Wf8TXQd/f3x/Lnh+KVq86I+Nkbzj0h5Gg9nVo3w83nnRC2WZIeb113\nJpo1jsEt551geBm8L5DT5IcGolnjWN64ERGR6ZjIS6LuY/c7Lzgx4ueNJAWdWusfB96K3KN108h9\nrBtrmISq9wkNR2mJlFfHxCjo1r5FxGVrMfqWc7BtxOXo0LKJoe/XbZ5zz8Du2DbicnxwyzmmxEZE\nRETux0ReEm9ccwZuO78bHh5yMm7vHzmRN2L8XX2j+r6RyueXrzy94XI0tMk3czZZK7XUcFOilZab\nl3D8Lh6YPkYBBvfsIDoMIiIiqTgjW/KADq2aYtRNvfHSFb3QKERC9/CQk2v/fmTI38IvMEiu3PuE\ntrprjxtF2XbkoYvrx6m1jfxpnVtHtV4nMLuphc8FbTcq/f6gr099ZCDOPP4YTcu4OYrmSVZy/t4h\nIiLZcBx5B3ly+Cno3qEl/nZsK3Rr3wJFZZWhPxwia+jarjmyi8pr/x1pSMcmGtqq66H1tqBb+xZ4\n7eoz8Nf2dDylcTx8QPuEU3pEWqIsg/5U+pyfKpb7gifygOLqJw5ERERGsEZeAp/dqa3JS4smjXB7\n/xPRv0d7w+v63/Vn1fv3sNM7hf18qJFu7PDAhT0w5eGBGPS3qpFmrEjStbAqfTS7At0fYYGX9gq/\nr2VQURn8N8hys0QUyvs39Ubca5eJDoOIPIaJvGAPX3wyLj+zs/kLDpH49D6hLUbecDYA4IqzOkcc\nEq9Jo+iHWdQqUmKrZTQfMQmfsZWafYPgi1Bj3aZ5Y5PXaL5yny/o6wqc3zTF7Jl8SS7/OL8b2rds\ngktOO1Z0KETkIUzkBft/Z3VGrFljGGp0xwUnInHUVRh/13loGiFRb2JyjbzVibaexddMktWjY8vo\n1mnwN51ucj+ASIm8E4SukVccP3yjw8MnjfjwyN0SR12FE9o1Fx0GUS0m8tTAKZ2Ojp0+NELTG73C\nzbwaabZULU1r9CTVjw3tiSXPXYI/nrwo/DK1L1KXs7pq67ypVcTOrg7IJB+5JHgnbgWRmw4RycDu\nihmy3+SHBogOgagWE3mXiuZSMv6uvujfoz1uOe8E3Ha+NUNhyqJ7x5ZCLrx6ZuXVKsSAL44S6umI\nG9rI8z7EG5o1tq85IolxQjtz5iIhMgMTecGaN7Gm0I8mZ+jZqTWmPDwQH9xyDmuXAMREyCJl2UKh\nhm6U1bOXnRr09WCduRUomtuYM2EmkXj4EZGdmMgL8NDFVePB9+/eHqd3Nl4za/cFw4xRY8KOI2/K\nD9IfY2yE33Vcm6boVV2DPjxIUyNZaou7HOOcdpvDTu+Ex4b2RM86zbjCURTnJ0hafys5nNMPVCJy\nFCbyArx8ZS8see4S/GxhOztJcssGZEl664qJUfDxbX1Cvq8oCib/awC+uPs8jL3j3IbvC97ajWIU\nNG0Ug9G3nCM0Dq1evaoXvr7vfMTEKLomHIu2jfxpx4mdZOyY5o3x9X39cNv53fDkcO1zIxARudnV\nvbuIDsHRmMgLIqptttsZvVG4rk9XHNemacj3j2nRGH8/szNaNDFnDrWOrUOv69M7+tY+AdBi1YvD\nsOal4RFrfO2uKAxVONd9sqM1N4/UvCmSsbefK8WMr8NOPw6jbuqta/8SEblZtOW71zGRJ2mY0bRG\n5uKgZ6dWUBSgSWwM3gmYmKuuq3p3iTiSTo3LzjgOndo0Q7uWTQAYn/TptvO74dJex+H2/t0MfT/Q\n9H8Pwns39TZlWQDQpJGCkztqa5oSOPrR+zf1xjXnHG9aLHoEa4oFyPlkiswRafQtIqqP5WF0mMg7\nmOwV+i9dcXqD16xuhtL7hLbo2tbetuJaC6Ghpx2Lpc8NxeqXhqFbe3NGPRh6Wv1EceSNZxtaTudj\nmmHivf3w7o3mJN8xiqJp9A6t265RTAzuGnASzunWFq2bNUKnME80At1ksCa+Zp6BaIS6gZD81CUi\nIodgIm+zR0OMk21EiyaNMLR6FkHZ2pjNe/piPDwkyG+1OIOJjVEw5ZGBGCO4vXiwZjqqCpzYoQU6\ntNKehEYSWPvXqXWzkJ+98uyjx8i5J7ZtEJvZQu3quqPPaO1A3ShWQZNGMfj934Ow4dVLMeRU7bNn\nGm3C9tjQnrh/cPiZjyMJVTtrRsdxkhNHTSLSh01rosNE3gZ96yRNw0yeYGnivedj7hMX4ZPbGnbC\nFOlUgR0Lu7ZtbqgW1ugFOFgRJHIs6SvP7hz09Ut7dcLTl56K6/oc3+B4EZV7aM2xG8VUFVWKoqBp\no1hD8Rpp8mBVM4nAGyk3+/KefqJDsBUTeSJ9mMZHh4m8zcxuDhMbo+CM49sgJmDBTqzx0zpOuGxE\nbutgTZXeuq5h+/s+3dpCURQ8eekp+Pi2c01r2hMtrTUxbusY3rFVU3x9Xz/cM/Ak0aFErUN1/4xQ\nzjZ5BmMS40RJygxyIXcV77ZjIk+28sr5Gux32nWb0jFI053Rt0Ro+y7oJirwBhQIvu0CE3m7wrVy\nPcNOPy7oTZfTNI7lZaTu8LV3DXD+zVkwke6lbzi3qz2BkOuIHsLZ6VgC28CZ9czWcOKTAplp3Zw9\nO9nb1ElRtMWm9YanUWxAIm/iWdXCotmVqYqeU/7BC6PrkwAE759itev6HE1iB/fsgP9dfxYeuvhk\nLHx2iO2xWCVS2f3+zeaNUkXy0jP3h1ZMC6LDRN52PGJDEX3DY/X6rajd1bLM/t3bm79ik2i5JlzQ\noz3aNGuseZn/v737Do+jOvcH/n216r1ZXbIk25ItuUsuknvBGEywjU3HxgaDHXqPacEpBFIIJFwI\nkIR2uYHAvSHcewMJhGYuJRASHm5+IXQTEriAqcYY1/P7Y2fl1Wpmtkzf/X6eZ5+VZmdnz5w5M/PO\nmTPndDck3pSjMDek+6B4gfaMg9XmXk5s868tHtobVERudmKH9IrCxPPTTRcsHh34k7qI4Ljpw3HR\nwWMwYlgxXr38IDRVBGfUZSPxNksm3JnRu9sJ7D9eZIL/PG2m7csM+C7vufTf8ygwAtpEXjfwCNqd\nB+8edo3ftOb2E6cmtczVfcMxc2Q16svycc+GvoHpeuVrXmfNkDRMaC7H3ev7hs7sE2Y9X+UnEMhf\ndfgE3HvKDNvSY2dRD9huk5CcUJbpxRcFx09X9+hOryjMMRwzIt1UxnkmJhXpuN+7yZ5hKokSlI77\nq5cHIbt+25HuJxNIXCIPu+ZlJ1fblRPKwh3rpkEplVAaYme579T9Qa7VbHGqbJTkZ2Pbl3uGTC/I\nDeEznenRUu1X30i8bRj76aiaYrz6/ueG8wb1gp7S36SWCt3pIoLFY+vw8N/edzU9R09tQXFeCL9/\n6X28uXW7K7/pxDGNbeStYY28C7w4Mfl1t7ByEPDrOunJ06kZ9WrEx0R+1+60JXxgTnWjJpDcRO+K\nmO2fVvddp/b9u9f3IVenKYMXt/iL85KrDyrOz8bYxlLdz4J2JytRbu/5nbUluGTJGHzLZARpsuaS\nJWMG/r70kC6smGzvBXIiDpvciIuXdGFYEgPkWZWee2iwMZB3WZDPU06n3euhzdf0tw78ffTUFkvL\nGtdYhtF1gx8wdXPQJa+ZbcvofPC8V8k4v79gjD9vl4+pL8Urlx80ZLpRjaGTivKSv3gwGvfCjuLg\nxxp9t7vWba0uxLpZ7agvNR4gLmle76s+s6pvOL61tBtXHT4Bi7pqdXvgclq6bJIgx0V+wECeEubH\nE2Si9GovY504sw0nzmzD0VNbsDGJNq1Gtc+/PtVaO+Rfnjwd7dVFpvPYtUm82rapjuiX2oBQyZvT\nMQxnL+wwHGQrHrdPUJd9pcvdH8TgHltihbJEt01t+7Bi3fndzK907q3Iif05J4vhQrS87BBW9bVi\nRU+TJ0G8Z5xoWpNB2ecE7pnkKq/217MOGBV3nvycEC49pAtXHDYOZTb06mF1dNdp7VV45Ly5OHth\nh+W0xGP3eT/RpjVzO4cN/D2qRj+481JkEK3rj9V/yC0euwOqqW3GPRDdcFwPygvNH0SLN3hTsr6/\ncjy6G/SbySyb2ID7Tp2B7JiLaLM8catpTUFOCIdOaHDlt7zgxHV5ug3K5oQJTe4OfuZFAOxMe3aW\nLSsYyLsgwBXZg9hx0DA7UTtZK+zkwzT6vdY49nODf8edn7FVdLObtTPasGR8PSa3lOMnxyUeLAd1\nFGCrogceipVImbthVWoXJEYO7202/OzcRZ0YqzOqq1+2XBCKUGdtauM/OFIjn2DXpolY1FVr27L8\nxO0i5cnzdzHHGavNUMk6BvIuC2Lg5RctVakPEe5k+3tu06ESvZDJCWXhumMm41enzMDIJGrk7dqa\nTm87O/vW/uMlC1FfZq0/8o4UA0NbORx9JLp0r5/JScSVK8bh9+fMHjJ97YxW19OSY2ON/E2re21b\nFrkrthQUp/CMTCy3bvak6104BvJpKh3bnHXUlmDDnBFory7CTTbXLAaV0XaeNap64O9EDl52x1Z+\nrO00SpPdaY0MMDWsJA8HdqfWtl6P0WA0Xkv2WONk0Yh+YN1MkI6P2Tpt00+fb95UMCdk/wqyaY37\nTps30vRzNeQP99nRHM6t/fGgsfYdj/3ElkBeRFaKyLUi8oSIfCYiSkTuSHFZTSJys4i8IyI7RWSL\niFwjIu53x2AXP0Y1HjHbXxPJpY0HjcYj583FoiQDJLf7qdX7PSeagxgt8nsrx2PJuHqs6W9N6NZn\nEGono/lpl4o9CV108Bj84qRpeOjs2QmPtJqOjE7Odm276Ds4o2qKceOqHlx4sP5D6ifPbrfnR10m\nIkPy8Z4NfXEH5XHi4ctMLstO23z+PDSWJ3+3ze3jYENZ/pDA3ey85rdRxRczkDd1CYDTAEwE8M9U\nFyIiIwA8D2AtgGcBXA3gDQBnAnhaRKqsJ9Vb6dpPcrRzDjB5ONOj1W+rTr1ZTjx621RvMzdXOpeG\nWPVlBbju2MnYdGj3kIcNI4qieu2YMtzeA26QirmVi5i71/dh8/nzBk3LEkH/iOq4D556wQ/bxa6L\nxp8cOxnVxXloKMvHbSdMxYHddYaDhx08rn5oOlwMguz8rSkJBEfZDgTyRmMUjK4rwe0nJDf6crpK\ndTu3VBViTtSD/xH74ixw4C6JS/v1z9dMSeqnfnHSNMfSkop0jb/sCuTPBtABoBTAVy0s53oANQDO\nUEotU0ptVErNRzig7wRwueWUkuPi3Q40YveJ9Were1FVlItFXbW2NnGIZXZo+NnqXhTmhtBVX4rV\nfa0pLd8s8LFyXLrr5D70DK/Amv5W3/aXbiSVotKq84xFY4W1NueVRbmuXqAly51aVP1CaHTSjLef\nz9UJaPSMqi3BUxvnY/MF89CQSm1m0t9wX6q7t93NYDYeNBp5BoH8b8+ajdkdiW2zdGflIlVvi+2L\ns7iB7exCYR7fVIYx9aVDzjlG+/nwqkLDSqRYyd4xr7NzfIQ0YMtRXin1qFLqVWWh7YCItANYBGAL\ngOtiPr4MwHYAq0TEvGNtckyiQaNf+tRd2FWL5y5eiJtW9zp7JW6y6IVdtXj+kgPwmzNm+u7W9Lim\nMvzHV/ux6dBuV2sqnKwJHW7yQPTisXWYP3r/BUt9WT5Onz/KUrMrvWzzupnSv564v3b02qP1B17y\nsysPG48JzeWY3FIed97c7KykurdMZT47pFomUt0tIzXydq3ihjkjkv7Omv5W016WnLDxoMFNq8pt\n6EbYLXrbKl5IFbL5uP2rU/oNP4v0KuZmM1Wj1btjHe8ARUtubG1nzdfeH1RK7Yv+QCm1TUSeRDjQ\nnw7gYbcTZ8WKnib0jww/fFjj4lDKdrPjxOd2W3U/XFQUODjwjNv5qef4vuG47em3vE4GAOBGk4eg\nRQQ3r5mCvfsUPtuxG8X52cgJZVkKvP3UVj9i1qhhA+3zY4NcN0uL0W/Fy7O6snzcpw2m1rrxN5bT\n0VCWj7EGfd2nq0RrQp00uq4ESyc24sy7XnDl97rqS7Fhzghc+cDfBqatmj4c1z7ymiu/74RhceIF\nO8foaqksxGSTkaEH2vAneBCx49g4t2MYHn35g0HTLl8+Fu3V/htzxEve7+37dWrvrxh8/qr2Hnd0\nHBF5Xu8FIPHhOm20uq8VX1s8Gl9bPDqlW8Cp8EOAp8e8AsGHUVGKZkb1GuMkr2t/T58/EqfOT60p\nlRVGNVWJHOBDWYKKolxbu4Z02qYERmyNzpJRtSUYXuXPm5dul9hfru/TvaCxuu9EHrY1G6TLqsqi\n3JSO5Qu1pnL+PAtYp3en6abVwe7JLHZbtVUXYVXfcNPvuN1G3m1rZrQN+v/SQ7pwjE7nDZneo5Kf\nzmSRkUM+Nfg8Mj3+/VZyJMBL0+dELNPLloPHDn24Lh2du6gTRbl+urHnLt2mNQ5EqkdMMR54yWtG\nxwXjXmv0M8i+dtaDl2/4DIPF7XTbCVNx+fKxuO6YyfFTlMJvrZvZhqaKwpSO5fM6tUA+DY7Zd540\nfeDvutJ8/PLk6QNdvEYsndiApoqh29nO1W9J4FkYO/f9358zx/Dh7YisyAa24255gpmV6HzJlFu9\nZV6+fCxmx1SITW2tNOzJKZMF6Qwc2XRxS4dSSvfSXKuVj3/UJbLIvZFd0+BMnQI7Y+VaCw9OudW0\nRq8v8Vi6Iww7qVWg0gAAIABJREFUkJZEJVs23R4bwuqmaywvwLHTzGtMU/XtZWNx3PTUl51OvXP0\njajCA2fOwlOvf4ilExt0x1MwbMZlw+9fsLgTL779Kc47sDP+zBbEbrJEapn9UBPt1Cjbyexb7dX+\nvPvoFj/VyEdq3IeO6R1WGjMfmfBrgGfaj3xAW9bodz/pz/x3QrInEzs284LR9vWys2HOCNSU5CFL\ngB8H8OHQCL39JyumHPqhXOqls7uhFPkGvaKkg2SObZVFuTiid/8dmESO5cneFXGDnb89pr4UJ85s\n82RQtFPmjsQNq3qSGnk6Famc/+x+2DUVSgHXHzu0ftSJ87kPVteX/FQj/7L2btQGPjKUnVEbegq4\ngMbxKC9wtmeEZp1bxgM8PLDVloZPqvk5IfSPqMJTr3/o2m8vm9iIF//xKbZ8uB2PRT0MlcqBvigv\nG098bR4++WJ30rXz+r3W2C/VE5gTI3zGMvqFdDrp5mVnYeeeffFntMGj581NuoercJt/venebQQ/\nVCa5nQK3K6NK8rUQzsUVjV1HBaC31dp4nYLwaORPvLo1xe97X9a85Kca+Ue190UiMihdIlICYAaA\nHQCecTthZK5neOI7cTqd3CO+u3K8o8tfNqkRU1orUJgbwk9iaz48vPr56eregb9vWTtl0GdOb+es\nLMGmQ7tx69rB3ZCleiLNyw5ZamLjtESyM08n+PNLl6ffWT5u4O8rVowzmdO6cw7Y3wTijAWjDOeL\nbRIwu2MYphgEJM9dstCVrhQnNJWhzM6KgTQ83qarVI6ZVR7coYjVPsyeZi1rZ7QO/L1icpMty8wU\nrtfIi0gOgBEAdiulXo9MV0q9LiIPItzF5KkAro362jcAFAG4USm13c30BpUTgZTRVe+irlq899mX\n+MfHO0xPnOlqxLBiPHzuHCy46nFHlh/KEtyzoR879+yN+/CTm8Y37X/u3E/p8lqybUa/t3I8bnj8\ndbzxQfKHtrUzWnHLk1vQVFGg+7BobM88XnU/eXhvE4ryQigvzMWkZmf7Kzi8twmf79yNHbv24aTZ\nbYbzxW6l20+Yim1f7sa4TQ8Ombc0P8dSe+RES0RZiqMBVxblYevnO1P6bjrwQ5MxwLt6FTvWPtVl\nHNnbjI++2DVkerKVKvM6a/DNpd1455MvsX52e4qpyUy2BPIisgzAMu3fyBCafSJyq/b3VqXUedrf\njQBeAvAWgNaYRZ0C4CkAPxaRBdp80wDMQ7hJzcV2pDcTuHmLL5QleOjsOXhj6+foqjfvr9nsFphT\nD824odGFbkV1g2V/nL+GMN/Ocb5rYZ18cj5PyhG9zTiitxnjLvsdtu3cozuPUaBy6ZIuHDyuHmPq\nS3UDzdzYwZKsJzclOaEsLJ3YaPh5sru+WZOhnFAWTp5tPoCRiGBsQxnue+Gd5H7YR769bCwuve8v\naK0qwoHddbjh8deHzBPA3SElY+pL9D8I4gEhBa6uZsy+mh3Ksty0JdwTjeiOfl5Xmo//++xLAPtr\n/4ccDzNjMxuy677rRADHa68DtWntUdNWJrIQrYa+F8CtCAfw5yJce/9jAH1KKfca4VJSCnJD6G4o\ns1QzEtwwPmPOFwnzapCl2Ic7vZBy+k2SbvRRVpZgSmslivP062RiB0Tbs9f+dt6G+7yDm+KpjQss\nL+P4/lb0tVehviwf/651X2d2/LISrFippDBK0nHTh+PpjQvw0NmzDS9s/FJTrefKw8ZBJPVKkHs2\n9KG2NA/T2iqxpt/gzkuAK4eS4WYb8ZALz91E+7eTpuGkWW24e30fCjO4q2MztuSKUmoTgE0JzrsF\nJod4pdTbANbaka5M5tfjt1/TRcH1zaXd+OkTb+CEGW2+6I4tZSYxh137jV/azMdKdv3ijXiZiNzs\nLNx58nQopXwT8J48K7kmBXVl7j3XkUgOHT21GXc++zZK8rOxZLz5WBpHTW3B7I5hqC7OQ8clDySd\nnimtlXh64wJbR+/+7opx+N5vX8aH24c2FUmE03eVRfSvTdwsvsV52VjUVYsH//oeVvYYt2W3Ky9G\nDCvGxUviD4iXyXh5Q5ZlSKUHJcFKDVGyJ6XVfa26t2Qp7OojJ+D6R1/HMdNaXK3RSqYMeHkMiQ7i\nvQ7nZ4ysGjKtoii1dvOAveuTyCa6eEkXJrdUYFJLOYoM7hJFszrSuZ1BPAAcOaUFdz77dsqBfDJK\n87Px2Zd7sKa/FYC7x0yrblzVgze3bkebSf/tDAvcw0CeXGV2wOEFQWZwYtRhr7jV97mVGuPlk5qw\nfBJ7gbDKSrCUaInX287Fedn4/srx+I8//QMnGzwE6JcuQIvzsnF4r49GIfbJnZZYU1srcce6aXj1\n/W1xnyuLlp0l2L13aGmyo2lNMscYEUH7MGf71U+GTzeza/x5n5UCZcbI6vgzkSMy/PjlOTcecg6q\nZE6ufjkR+yUdsQ7vbcZdJ/dh/uha3c+NervJ9P619azpb8WsUcbnLDfKwC9Omobc7KxBz5WdOm8k\nIjcYTp8/cmDe7ywfh/bqInxzaXegmg5GKubOP7AT+TlZA3ce9FjN80yvBGSNPKVs6cQG9LZWoqsh\n8RoFhp6U7soLnR0gLB25fSJ2+yg0vNJkUDcb9EaN5eFUIGrXYu0clTlZZQU52HRoN+798z9SHnzI\nDtmhoXWodWX5uPeUGXjt/c8HPV9wzLQWHDOtBQDw7f9+SXd5frwAjdx5PXXeSKyf3Y59Crj1qS3e\nJipNMZCnlP3oKHuHsw9y95OUuHTczDet6sGNm9/A4T1NKMlnIA84FyyPsGkAGjdNa6/CYZMbsfmV\nD7D1c/vbX09oLscZ80fi0Zc/wIUHjR6YbmeAZ8duW12ch8uXOzsgmBmnA16rx7YJzeWYYDLOwq6o\nXqeyfV47vyeqCVB2KAu7HBwZ2Y8XMm5iIE8Js2NnyfQdLlNY2c5r+tvwzBsfAYBprwh+sqi7Dou6\n6+LPmIbsuABPtLwcNaUZ6+eY9xFvKR0mlx9WD10/PGIilFJou/B+i0vSd86iTpyzqHPQNL8dbi9Z\nMsbVnnZiycC7c9vZLcX5+8M3v/S6FC02cDdLot1NwM6YPxIzRlbjwnv/N6WB9oKGbeTJN06fn3mj\nwtJQB3bX4tJDurB+djsuWTLG6+RQHEZhfDLBRSLXAjUlebhyxXjTnjL8zo8BV6K8SHlZgTN3t5x6\n4H7xWGcv5s89oGPg7wsOjLrzYsOy7d6+Ox2sgY8nO5SFae1VGTPiOGvkyVUdNfpPup+xYBRW9w93\nOTX2yc7y5po4yIGBERHBiTMNBnghT+RlZ2FFTxN+8Ye/u/q7PcMr8PxbHwMA5nQMc/z30m53Cvj6\n/HR1r6Xvu736X507Am9s3Y5PvtjlSBv8E2a2YdfefSjMzcaRU/b3DuTHcrsrZvA5N5Pow+xwFAN5\nctwdJ07D6Xf+CaNqSnDs9KHB+tFTW3BOVE1DEIWyBN9c2o1bntyCk5Ic1IXIr2aNqsb5B3airiwf\ndzyjH8Qb1abbcTK95siJ2HDH8yjKzcbFHt+d8WOwFE9Qe635+fG9aK4sREdtia3LjVR8OJUv+Tkh\nXHt0+Nmx1o2/GZheVZQ70De9WRv4eIrysnFuTPMpv5rb6fyFd0QwS7l9GMiT42aOqsZzFy/UfVI/\nnXBgov2CGPR47dJDunDBf7wIADhl7ghc/9jrHqcoHPiMb4oXeDj39HJzZSF+c8Ys+xfM8ulr80fX\nOHK3MZElOvG7PzhiAi659y9QSuFHR060ffl+K86j60qGPNTs5N3jNOw/ISkM5MkVZkE8gz4i4LDJ\njdixey927dmHQyc2+CKQT+RBVsMaee7XngviNjiit8nWoC83lDXQzGNYSR4A9/OlqbwAmy+YB6WU\nIxVafmti+cCZs5IcYMre348sz1+54pz0riLNYH7bsc0EJ6WUyaL76nZCdigLx/e34qTZ7chNg7tX\nQW3WYSx46xO8FMP25jS3nTAVIuHuGs26THa6O8dQljh2V9qWlNu4+nrxh9nirf507PeDFP/YIfhn\nCwq8DNvnbOVm1p1/YKfu35nimqPsvyXud9EnRKOylk63tc2ORTOjRgO1q/1vYa6zvWpkWkCjp29E\nFTafPw9PbpyPzjrji4RDJzYACObFT9A38/LJjY4sN+j5kig2rSGihKyf3Y6migJ8umM3juhtjv8F\nA0Ed+KupohDnH9iJ7//uZcd/yy85ZKVpTSAjIhPFedn4r9Nm4pk3PsSySfYEHoeMr8enO3bjL//8\nDN9fOd6WZTrFyTKZJcA+7QcmtaT+MKiRZodH1vWe/3e22KD62YsX4OdPvImuhlJ0N5Q58psBPdUk\njYE8Jcz/hwpykohg6cTEApj0a1bhLj9e7GRC7Va8cjuuqQzjmuwLOrJEcOOqXiilnHm408ZF7nOw\nTP7qlBn4zv0vYVpbJXqGV9q23Ewos0Aw17OmJB8XHmxPT1Sx+07kXyfLrJ8wkCfPMegLBm4l0mN0\n0eHn4MJvSXOqCYzf1tPIxOZy3L2+z+tkOMbP+4JbRAT3ntKPX/3pnzjM5qY0fqz4cBPbyFPCLj2k\na+DvjQeNNpkzOTzIEQ0WPaJlcZ6/61vS6RSaE3L+YLSyp2ng71V9ARoEL402tFlNbYHDzy04wY8j\nu+qZ1FKBby0bi0kt9ncc0N1QOvD3wjG1AFgjTwHnxE45f3QNrj5yArZ9ucdSG+lYjONTx4ug9JQd\nysJ9p87Ab/73XayY3BT/Cx5yckAoOzVVFOAfH+8AAMOHHkUEP1vdi3W3/9GxdFyyZAxaqwoxsqbY\nsbbBEUFpWuO2mEFHB/nW0rGYd9VjUAr4weET3EuUBTwPAD85tgc3bn4dk1sqBno+2pc+RdYUA3lK\nmIhg+SR/BxVETsrPca+2bkJzuaVRIO1QVZQ78HfQm8DdtKoXR970NLKzxDRAW9hVi4rCHHz8xW5H\n0lFemIvT5o9yZNlDBXubOWXvPuNIvrW6CA+fMwcfbt/leJezbmgoy8fo+lJMa6vEFQ/8zevkOEJE\n0FJVOGQQqnS6+DTDpjVp5ICu2oG/V/QEJ+BmF2nJqYwKrnpcPNEks5nM5g3ysfWYqS0DzV7OXOBW\nMOaujtpiAEBBTggXLYn/MJoyaHMRb78e17i/NtqN4dy7Gkrxh4sW4OkLF2B4VZHpvOlyTLJzNYK8\n38Yyq5EHgPZhxZjSWmmpHIzW7vrkhAQtleblzaosk3R+ZWIDbl4zBdPbqxxNgx+lU5k1wxr5NHL5\n8rEoyAmhsigXq4PU9pKScudJ0/GTx17DjJHVcQMSsldBbgiPnDsHr73/Oaa02te7hp/852kz8d5n\nX6K8IBdlhfvb6teX5+t/IcWT5bVHT8I5d7+AyqJcnO5SDXVhLk950QpzQziwuw5/fOsjvP3RDtN5\ng1i7aXQXyaxG3i4/O74X973wDuZ11iA329k6U7PrjUiQH7ytZ10Qy2wqeFRLIzUl+fjx0cYj1/nJ\nAV21eOiv7wEY/PAXxddZV4JrTEYoJGdVFeehqjjP62Q4Jj8npHuBeNikRlzw7y8OmZ5qN/Kt1UX4\n1Skzkk8gJcxsG/zbummY2laJnFAW3ty6HXc993csGF2Ltbc8i+279g6ZP51Coj0uNJ5uqijEqfNG\nOv47gHmzt5NmtQOI37NLutyFisZAnshB31k+DiNrijG6rgRjG5194IuIrMsOZeHWtVOw5pbnBk3P\nkHNlIJkFZ/0jqgY+b6suwoUHjTH9Tjpt570Z8hTk2MbSgaaYmbHGg6VTmTXDQJ48MawkD19bbF8X\nlkTkPL0gz7iNvNOpISuSr4FNn6go7QJ5g005qmZ/70yZEtRGy5R15sOuRGSoJC8bKyY34a6Tpyd1\n4jebM0OOrRkjHU+W0Q/itlUH9zkUO6+l0mk7702nlYHxdh7cnCa91jma0fpnStMaBvJENEh1VPvv\nU+aNxFVHTMjIHg/ImiB3V3nFYeNQX5aP6uJc3Liqx+vkpMzWXmtsWEYQB1tKFxkS0w7CQJ6IMtLt\nJ0xFR20x5nUOw4kz22xffnT3mRR86XiqbCgvwBMXzMNTGxcMDC6TKRKr3U1cpJvWtuqigRE33WJ0\nIXPs1OHIz0mf8Mes+8lEBfey21i6taAywjbyRDRIV0MpHjx7jiPLbq8uwpFT7BsVmNylF8ylGuD5\nXXYo+IGenXdFUt3KZx/QgSXj6zG8qhChLH+Ei2WFOfiv02bigKs3D0yzIxj2SiKVI+m5l5pL12NT\nrOAfqYjId/TOiXev78ODZ89GThoESLRfZpwqg8kvA0J11JYgL9tfzWpG1ZYMdLgQyhKcu6jD4xSl\n7qyFo1CcN7RedlALeQd31Ori3IGB5PpH+KcZJmvkiYhsVFOSlxa1nOkiNzsLu/Y4PzAOeae5ojD5\nLxkE/0GJifpHVOGp1z8EABw8rt503nWz2tBWXYiWyiLUlxW4kTxHlBfm4smN8/HBtp1Y+MPHdedp\nrnRu/UQEd6ybhide2Yr5o2sc+51ksUaeiIjS1q++2o/lkxoxobnc2oIy41wZSGWFOfiXYybhoLF1\nlpcVlKDoh0dMxPrZ7bjumMkYWVNsOm9OKAuLx9ajq6HUpdQ5p6wgx3R968sKcMmSMZjUUo5b106x\n/fdrSvKxoqcJFT56BipTauQZyBORKwLcBDUtjW0sw9VHTsTBFoM8o37kyR8OGd+AnxzXM6g3qnRW\nV5aPCw8egyXjzWvjM9G6We2495QZmNs5tNY8yMdno7Rfedi4gb8vXz7WpdS4j01riMgVQe6OkMKS\nCdkZ4KeXgFTIU5RM32aLuutw9ZETsHP3Phw2ucnr5DiGgTwROYBBe1BYrYkzChaC3AtIJpvTMQz/\n/eK7Q6bzwoyCJpQlWD4pfQP4CDatISLKYAU51noT0Qvkl4yvR77F5ZK9Er2u2nRot+70TK/dDSK7\nNllNif+aZU1qCT/b01lbYvkYFnSskSciV7CC1p8O723GDx96BR9/sRtnzB+Z9Pdjg4UnLpiHporg\n9gCS6aqL8zBrVDWeeHXroOl6gfyRvRwTIt1VFuXi5jX2Pxxr1Y3H9eDBv76H+aNrIBl+cmEgT0SU\nwfJzQnjk3Ll45b1tmNJaaXl5zZUpdHlIvqIXtMc2rfnO8nE4ZAIfKE0Xes8wddWX4t5T+303BgAA\n1JTm47jpw71Ohi8wkCciV+gNWEL+UFGUi2ntqQ3kEpRuCcma2M18zLQWbxJCrskJiS+DeBqMbeSJ\nyHaRO51fP6QLpfnZ2DBnhK/6F6YU6dbUUhBYbXzA7Rw8ZhfZ0V0zGsn0JitBwUCeiBxzwsw2vPD1\nRdh40Givk0KU0SzHZIzk08pRU/XvqEQPEHdAV61bySELGMgTkaOyslirk87YsiYYGsv3P4CcSi8f\n+7ihM8K1R01CX3sVloyvx7pZbV4nhxLARqtERGQBA7wguOqIiTjoR5uxZ6/CbSdMNZ1Xr894buX0\nFrlj01JViDtPnu5tYigpDOSJiChlrKgNhrbqIvzhwoXYuXcvakryk/7+mv5WPP/WxwCAFWk8SmY6\n4a6ZGRjIE5Ht2JgmPenV1E5t299lZWdtiZvJoSSVFeYAyEnpu0vG1eOfn+zAB9t24rR5yY83QETO\nYCBPREQpqyrOw20nTMWTr23FcdPYr3O6ysoSbJgzwutkUBxH9jbjl398GwBwwgy2cc8EDOSJiMiS\nOR3DMKdjmNfJIMp4Fx08Bo0VBRheVYie4RVeJ4dcwECeiIiIBvC5h+AqK8zBGQtGeZ0MchG7nyQi\nIiIiCiAG8kRkO44ImJ5YU0tE5C8M5InIdpWFuV4ngYiIKO0xkCciW9y0qgeTWsrx7WVjtW7uiIiI\nyEl82JWIbLGouw6Luuu8TgYRWcQmVJmHzSGDizXyREREREQBxECeiIgSwppaIiJ/YSBPRERERBRA\nDOSJiIiIiAKIgTwRERFRBuOjrsHFQJ6IiIgGKPBhCKKgYCBPREQJYXhHROQvDOSJiIiIiAKIgTwR\nESVEsf9JIiJfYSBPRERElME4sGtwMZAnIqKEcBj3zMAbL0TBwUCeiIgSwqY1RET+wkCeiIiIiCiA\nGMgTEREREQUQA3kiIkrIjJHVyNKayc/tHOZtYojINnz8JbiyvU4AEREFQ1FeNv7ztJl45o0PsWxS\no9fJIYfwSYjMw8dfgos18kRElLCxjWVYN6sd1cV5XieFiCy44rBxA39/d8V4D1NCVrBGnoiIiAaU\nF+R4nQRyweE9TSjNz0FFUQ7GNpZ5nRxKEWvkiYiIaMBlh3YjJxRuNH3t0ZM8Tg05JTuUhSXj69E/\notrrpJAFrJEnIiKiAY3lBXjya/Px8Re70VlX4nVyiMgEA3kiIiIapKY0HzWl+V4ng4jiYNMaIiIi\nIqIAYiBPRERERBRADOSJiIiIiAKIgTwRERERUQDZFsiLSJOI3Cwi74jIThHZIiLXiEhFksuZKSL3\nad//UkT+LiL3i8hiu9JKRERERBR0tgTyIjICwPMA1gJ4FsDVAN4AcCaAp0WkKsHlfBXAEwAWaO9X\nA3gcwBwAD4jIxXakl4iIiIgo6OzqfvJ6ADUAzlBKXRuZKCI/BHA2gMsBbDBbgIjkALgCwJcAepRS\nL0d99h0AfwZwsYj8QCm106Z0ExEREREFkuUaeRFpB7AIwBYA18V8fBmA7QBWiUhRnEVVAigD8Ep0\nEA8ASqmXALwCoABAsdU0ExEREREFnR1Na+Zr7w8qpfZFf6CU2gbgSQCFAKbHWc77AD4A0CEio6I/\nEJEOAKMAvKCU+tCGNBMRERERBZodgXyn9v6Kweevau8dZgtRSikAp2ppel5EbhORK0TkdoTb3/8/\nAIfbkF4iIiIiosCzo418mfb+qcHnkenl8RaklLpHRN4BcCeA1VEfvQfgFoQfoI1LRJ43+Gh0It8n\nIiIiIvI7N/qRF+1dxZ1R5DgAv0e4x5oxCDfJGQPgYQD/AuAuh9JIRERERBQodtTIR2rcyww+L42Z\nT5fWDv5mAC8CWBXV3v5vIrIK4SY8h4vIXKXUY2bLUkr1GPzG8wAmm32XiIiIiCgI7KiRj/QwY9QG\nPvLgqlEb+ohFAHIAPK7z0Ow+AJu1f3WDdCIiIiKiTGJHIP+o9r5IRAYtT0RKAMwAsAPAM3GWk6e9\nDzP4PDJ9VyqJJCIiIiJKJ5YDeaXU6wAeBNCKcK8z0b4BoAjA7Uqp7ZGJIjJaRGIfPH1Ce18pIuOj\nPxCRiQBWItzO/hGraSYiIiIiCjoJ9/pocSEiIwA8hfDorvcBeAnANADzEG5S0x/d/7uIKABQSknM\ncm4GsBbhWvd7AbyF8AXCMgC5AK5RSp1tIZ0fFhQUVI4ZMybVRRARERERxfXSSy9hx44dHymlqpz6\nDVsCeQAQkWYA3wSwGEAVgHcB/BrAN5RSH8XMaxTIC4DjAawBMAFACYDPAPwZwE+VUpZ6rRGRNxF+\n+HaLleWkKHIH4m8e/HaQMd9Sw3xLDfMtNcy31DDfksc8Sw3zLTVW860VwGdKqTZ7kjOUbYE8mYv0\nbW/Uow7pY76lhvmWGuZbaphvqWG+JY95lhrmW2qCkG9u9CNPREREREQ2YyBPRERERBRADOSJiIiI\niAKIgTwRERERUQAxkCciIiIiCiD2WkNEREREFECskSciIiIiCiAG8kREREREAcRAnoiIiIgogBjI\nExEREREFEAN5IiIiIqIAYiBPRERERBRADOSJiIiIiAKIgbzDRKRJRG4WkXdEZKeIbBGRa0Skwuu0\nuUFbX2Xw+j+D7/SLyP0i8pGIfCEiL4rIWSISMvmdQ0TkMRH5VEQ+F5E/iMjxzq2ZdSKyUkSuFZEn\nROQzLU/uiPMdV/JGRI4XkWe1+T/Vvn9Iqutqp2TyTURaTcqfEpG7TH4nqTwQkZC2LV4UkR3aNrpf\nRPrtWG8rRKRKRNaJyL0i8pqWvk9F5H9E5EQR0T0XZHp5SzbfWN72E5HvisjDIvJ2VPr+LCKXiUiV\nwXcyurwByeUby5s5EVkVlRfrDOZxvPw4nndKKb4cegEYAeA9AArArwFcCeAR7f+/AajyOo0u5MEW\nAJ8A2KTzOk9n/qUA9gD4HMDPAXxfyysF4B6D3zhN+3wrgOsAXA3gbW3aD7zOA5O8eUFL4zYAL2l/\n32Eyvyt5A+AH2udva/NfB+BDbdppQco3AK3a5y8YlMGVduQBAAFwT9S+/X1tG32ubbOlHufZBi1t\n7wD4NwBXALhZ2zcVgH+HNkAgy1vq+cbyNiiNuwA8o+XXlQCuBfCcluZ/AmhmebOWbyxvpvnYrO2n\n27R0r/Oi/LiRd55ndjq/APxO23inx0z/oTb9Bq/T6EIebAGwJcF5SwG8D2AngN6o6fkAntLy7KiY\n77QC+FLbkVqjplcAeE37Tp/X+WCwvvMAjNJ29LkwD0hdyRsA/dr01wBUxCzrQ215rVbW2+V8a9U+\nvzWJ5SedBwCO1r7zJID8qOlTtG32PoASD/NsPoCvAMiKmV4H4O9a2lewvFnON5a3qLJiMP1yLe3X\ns7xZzjeWN/11FAC/B/A6woHzkEDerfLjRt55nuHp+gLQrm28NzH0JFCC8NXYdgBFXqfV4XzYgsQD\n+RO0PLtN57P52mePx0z/pjb9G8ksz28vxA9IXckbALdr09fqfMdweT7Ot1Ykf6JLOg8AbNamz0tm\neX54AbhIS9+1LG+W843lLf76TtDS9xDLm+V8Y3nTX8czAewDMBvhOxN6gbwr5ceNvGMbeefM194f\nVErti/5AKbUN4auzQgDT3U6YB/JE5DgRuUhEzhSReQZtHiN59ludzzYD+AJAv4jkJfidB2LmCTK3\n8iZd87NBRNZrZXC9iIw3mTepPNDyvB/hbfBEIt/xmd3a+56oaSxv8enlWwTLm7GvaO8vRk1jeYtP\nL98iWN40IjIG4SZJP1JKbTaZ1fHy41beZVv5Mpnq1N5fMfj8VQCLAHQAeNiVFHmnDsC/xkx7U0TW\nKqUej5pRMcxGAAAGIklEQVRmmGdKqT0i8iaAboTvdryUwHfeFZHtAJpEpFAp9YWVlfCY43kjIkUA\nGgF8rpR6VycNr2rvHRbWwysHaK8BIvIYgOOVUn+PmpZKHowEEALwhlJKL6jzbb6JSDaA1dq/0Scn\nljcTJvkWwfKmEZHzABQDKAPQC2AmwsHolVGzsbzFSDDfIljeMLBf/ivCzd4uijO7G+XHlbxjjbxz\nyrT3Tw0+j0wvdyEtXroFwAKEg/kiAOMA3IjwLcEHRGRC1Lyp5Fmi3ykz+Dwo3MibdCyzXwD4FoAe\nhNs+VgCYA+BRhJvlPKwdoCOczGc/5tuVAMYCuF8p9buo6Sxv5ozyjeVtqPMAXAbgLISD0d8CWKSU\n+iBqHpa3oRLJN5a3wb4OYBKANUqpHXHmdaP8uJJ3DOS9I9q78jQVDlNKfUMp9YhS6j2l1BdKqb8o\npTYg/MBvAcLt1xKVSp5lRD7D3bwJTF4qpd5XSn1dKfUnpdQn2mszwnfD/oBwjYlut2TxFp3EvL4s\ngyJyBoBzEe5JYVWyX9feM668meUby9tQSqk6pZQgXJlzGMK16n8WkclJLCbjylsi+cbyFpUIkakI\n18JfpZR62o5Fau9Olh9b8o6BvHPi1QSXxsyXaW7Q3mdHTUslzxL9zmdJpc5/3MibePPHq10IDO02\n58+0f5Mpg3p5ELh9XUROBfAjAH9F+CGsj2JmYXnTkUC+6cr08gYAWmXOvQgHmVUIP+gXwfJmIE6+\nGX0no8pbVJOaVwBcmuDX3Cg/ruQdA3nnvKy9G7V9GqW9G7WhT3fva+/Rt/0M80zbUdsQfrDsjQS/\nU68t/x8Bbx8PuJA3SqntCPdTXKx9HivdymzkFvVAGUwxD14DsBdAu7YtEvmOZ0TkLAD/AuAvCAej\negOzsbzFSDDfzGRkeYullHoL4QuhbhGp1iazvMVhkG9mMqm8FSNcDsYA+DJqECiFcPMkAPipNu0a\n7X83yo8recdA3jmPau+LZOjofyUAZgDYgfDAD5moT3uPPjA/or0v1pl/NsK9/DyllNqZ4HcOipkn\nyNzKm0zJT2B/j1FvxExPKg+0PH8K4W0wK5HveEVEvobwICYvIByMvm8wK8tblCTyzUzGlTcTDdr7\nXu2d5S0xsflmJpPK206EB1nSe/1Zm+d/tP8jzW4cLz+u5Z2Vviv5ituXaUYPCIVwLwOVOtOHI/y0\ntgJwUdT0UoRrEZIZFKQNAR0QKmY95sK8P3RX8gYBGDAlyXybBiBXZ/p8bV0UgH6reYDEBv0o9Tiv\nLtXS+Ee9/ZLlzZZ8Y3kLp2M0gDqd6VnYP7DRkyxvlvON5S1+nm6Cfj/yrpQfN/LO80xO5xeAEQDe\n0zbirxEe3vsR7f+XAVR5nUaH13+TVrAfAHA9gO8iPKT5Di0PfhN7EAKwDPuH6f4ZgO8haphuxAwj\nr33ndO3zhIdZ9sNLW9dbtddvtfS+HjXtBzrzO543AK7SPo8egnqrNs0PQ5gnnG8AHkM4QLhHW5er\nEe7uVWmvS+zIAwwehvslbdv4ZghzAMdradujrc8mndcaljdr+cbyNpC+sxDuZ/9hADchfO67GeH9\nVAF4F0AXy5u1fGN5SyhPN0EnkHer/LiRd55ncrq/ADQj3AXjuwB2AXgL4YelTGt20uGFcDdYd2oH\n40+0A9QHAB5CuA/mIQdm7XszANwP4GOEg/7/BXA2gJDJb30FwOMAtiE8Yu5zCPeh63k+mKQ5coAx\nem3xKm8QDmCe0+bfpn3/EK/zLNl8A3AigP9GeIThzxGuAfk7gF8CmGVnHiA8LsfZ2jbZoW2j+xFT\nI+bTPFMAHmN5s5ZvLG8DaRuLcIDzAsJBzh6EH+h7TstT3fMfy1ty+cbyllCeRvbhIYG8W+XH6bwT\n7UeIiIiIiChA+LArEREREVEAMZAnIiIiIgogBvJERERERAHEQJ6IiIiIKIAYyBMRERERBRADeSIi\nIiKiAGIgT0REREQUQAzkiYiIiIgCiIE8EREREVEAMZAnIiIiIgogBvJERERERAHEQJ6IiIiIKIAY\nyBMRERERBRADeSIiIiKiAGIgT0REREQUQAzkiYiIiIgCiIE8EREREVEA/X/7Ev/ap+qJVQAAAABJ\nRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1a14f87a20>"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 250,
       "width": 377
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(losses['test'], label='Test loss')\n",
    "plt.legend()\n",
    "_ = plt.ylim()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 获取 Tensors\n",
    "使用函数 [`get_tensor_by_name()`](https://www.tensorflow.org/api_docs/python/tf/Graph#get_tensor_by_name)从 `loaded_graph` 中获取tensors，后面的推荐功能要用到。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_tensors(loaded_graph):\n",
    "\n",
    "    uid = loaded_graph.get_tensor_by_name(\"uid:0\")\n",
    "    user_gender = loaded_graph.get_tensor_by_name(\"user_gender:0\")\n",
    "    user_age = loaded_graph.get_tensor_by_name(\"user_age:0\")\n",
    "    user_job = loaded_graph.get_tensor_by_name(\"user_job:0\")\n",
    "    movie_id = loaded_graph.get_tensor_by_name(\"movie_id:0\")\n",
    "    movie_categories = loaded_graph.get_tensor_by_name(\"movie_categories:0\")\n",
    "    movie_titles = loaded_graph.get_tensor_by_name(\"movie_titles:0\")\n",
    "    targets = loaded_graph.get_tensor_by_name(\"targets:0\")\n",
    "    dropout_keep_prob = loaded_graph.get_tensor_by_name(\"dropout_keep_prob:0\")\n",
    "    lr = loaded_graph.get_tensor_by_name(\"LearningRate:0\")\n",
    "    #两种不同计算预测评分的方案使用不同的name获取tensor inference\n",
    "#     inference = loaded_graph.get_tensor_by_name(\"inference/inference/BiasAdd:0\")\n",
    "    inference = loaded_graph.get_tensor_by_name(\"inference/ExpandDims:0\") # 之前是MatMul:0 因为inference代码修改了 这里也要修改 感谢网友 @清歌 指出问题\n",
    "    movie_combine_layer_flat = loaded_graph.get_tensor_by_name(\"movie_fc/Reshape:0\")\n",
    "    user_combine_layer_flat = loaded_graph.get_tensor_by_name(\"user_fc/Reshape:0\")\n",
    "    return uid, user_gender, user_age, user_job, movie_id, movie_categories, movie_titles, targets, lr, dropout_keep_prob, inference, movie_combine_layer_flat, user_combine_layer_flat\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 指定用户和电影进行评分\n",
    "这部分就是对网络做正向传播，计算得到预测的评分"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "def rating_movie(user_id_val, movie_id_val):\n",
    "    loaded_graph = tf.Graph()  #\n",
    "    with tf.Session(graph=loaded_graph) as sess:  #\n",
    "        # Load saved model\n",
    "        loader = tf.train.import_meta_graph(load_dir + '.meta')\n",
    "        loader.restore(sess, load_dir)\n",
    "    \n",
    "        # Get Tensors from loaded model\n",
    "        uid, user_gender, user_age, user_job, movie_id, movie_categories, movie_titles, targets, lr, dropout_keep_prob, inference,_, __ = get_tensors(loaded_graph)  #loaded_graph\n",
    "    \n",
    "        categories = np.zeros([1, 18])\n",
    "        categories[0] = movies.values[movieid2idx[movie_id_val]][2]\n",
    "    \n",
    "        titles = np.zeros([1, sentences_size])\n",
    "        titles[0] = movies.values[movieid2idx[movie_id_val]][1]\n",
    "    \n",
    "        feed = {\n",
    "              uid: np.reshape(users.values[user_id_val-1][0], [1, 1]),\n",
    "              user_gender: np.reshape(users.values[user_id_val-1][1], [1, 1]),\n",
    "              user_age: np.reshape(users.values[user_id_val-1][2], [1, 1]),\n",
    "              user_job: np.reshape(users.values[user_id_val-1][3], [1, 1]),\n",
    "              movie_id: np.reshape(movies.values[movieid2idx[movie_id_val]][0], [1, 1]),\n",
    "              movie_categories: categories,  #x.take(6,1)\n",
    "              movie_titles: titles,  #x.take(5,1)\n",
    "              dropout_keep_prob: 1}\n",
    "    \n",
    "        # Get Prediction\n",
    "        inference_val = sess.run([inference], feed)  \n",
    "    \n",
    "        return (inference_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[array([[ 4.27963877]], dtype=float32)]"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rating_movie(234, 1401)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 生成Movie特征矩阵\n",
    "将训练好的电影特征组合成电影特征矩阵并保存到本地"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "loaded_graph = tf.Graph()  #\n",
    "movie_matrics = []\n",
    "with tf.Session(graph=loaded_graph) as sess:  #\n",
    "    # Load saved model\n",
    "    loader = tf.train.import_meta_graph(load_dir + '.meta')\n",
    "    loader.restore(sess, load_dir)\n",
    "\n",
    "    # Get Tensors from loaded model\n",
    "    uid, user_gender, user_age, user_job, movie_id, movie_categories, movie_titles, targets, lr, dropout_keep_prob, _, movie_combine_layer_flat, __ = get_tensors(loaded_graph)  #loaded_graph\n",
    "\n",
    "    for item in movies.values:\n",
    "        categories = np.zeros([1, 18])\n",
    "        categories[0] = item.take(2)\n",
    "\n",
    "        titles = np.zeros([1, sentences_size])\n",
    "        titles[0] = item.take(1)\n",
    "\n",
    "        feed = {\n",
    "            movie_id: np.reshape(item.take(0), [1, 1]),\n",
    "            movie_categories: categories,  #x.take(6,1)\n",
    "            movie_titles: titles,  #x.take(5,1)\n",
    "            dropout_keep_prob: 1}\n",
    "\n",
    "        movie_combine_layer_flat_val = sess.run([movie_combine_layer_flat], feed)  \n",
    "        movie_matrics.append(movie_combine_layer_flat_val)\n",
    "\n",
    "pickle.dump((np.array(movie_matrics).reshape(-1, 200)), open('movie_matrics.p', 'wb'))\n",
    "movie_matrics = pickle.load(open('movie_matrics.p', mode='rb'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "movie_matrics = pickle.load(open('movie_matrics.p', mode='rb'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 生成User特征矩阵\n",
    "将训练好的用户特征组合成用户特征矩阵并保存到本地"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "loaded_graph = tf.Graph()  #\n",
    "users_matrics = []\n",
    "with tf.Session(graph=loaded_graph) as sess:  #\n",
    "    # Load saved model\n",
    "    loader = tf.train.import_meta_graph(load_dir + '.meta')\n",
    "    loader.restore(sess, load_dir)\n",
    "\n",
    "    # Get Tensors from loaded model\n",
    "    uid, user_gender, user_age, user_job, movie_id, movie_categories, movie_titles, targets, lr, dropout_keep_prob, _, __,user_combine_layer_flat = get_tensors(loaded_graph)  #loaded_graph\n",
    "\n",
    "    for item in users.values:\n",
    "\n",
    "        feed = {\n",
    "            uid: np.reshape(item.take(0), [1, 1]),\n",
    "            user_gender: np.reshape(item.take(1), [1, 1]),\n",
    "            user_age: np.reshape(item.take(2), [1, 1]),\n",
    "            user_job: np.reshape(item.take(3), [1, 1]),\n",
    "            dropout_keep_prob: 1}\n",
    "\n",
    "        user_combine_layer_flat_val = sess.run([user_combine_layer_flat], feed)  \n",
    "        users_matrics.append(user_combine_layer_flat_val)\n",
    "\n",
    "pickle.dump((np.array(users_matrics).reshape(-1, 200)), open('users_matrics.p', 'wb'))\n",
    "users_matrics = pickle.load(open('users_matrics.p', mode='rb'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "users_matrics = pickle.load(open('users_matrics.p', mode='rb'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 开始推荐电影\n",
    "使用生产的用户特征矩阵和电影特征矩阵做电影推荐"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 推荐同类型的电影\n",
    "思路是计算当前看的电影特征向量与整个电影特征矩阵的余弦相似度，取相似度最大的top_k个，这里加了些随机选择在里面，保证每次的推荐稍稍有些不同。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "def recommend_same_type_movie(movie_id_val, top_k = 20):\n",
    "    \n",
    "    loaded_graph = tf.Graph()  #\n",
    "    with tf.Session(graph=loaded_graph) as sess:  #\n",
    "        # Load saved model\n",
    "        loader = tf.train.import_meta_graph(load_dir + '.meta')\n",
    "        loader.restore(sess, load_dir)\n",
    "        \n",
    "        norm_movie_matrics = tf.sqrt(tf.reduce_sum(tf.square(movie_matrics), 1, keep_dims=True))\n",
    "        normalized_movie_matrics = movie_matrics / norm_movie_matrics\n",
    "\n",
    "        #推荐同类型的电影\n",
    "        probs_embeddings = (movie_matrics[movieid2idx[movie_id_val]]).reshape([1, 200])\n",
    "        probs_similarity = tf.matmul(probs_embeddings, tf.transpose(normalized_movie_matrics))\n",
    "        sim = (probs_similarity.eval())\n",
    "    #     results = (-sim[0]).argsort()[0:top_k]\n",
    "    #     print(results)\n",
    "        \n",
    "        print(\"您看的电影是：{}\".format(movies_orig[movieid2idx[movie_id_val]]))\n",
    "        print(\"以下是给您的推荐：\")\n",
    "        p = np.squeeze(sim)\n",
    "        p[np.argsort(p)[:-top_k]] = 0\n",
    "        p = p / np.sum(p)\n",
    "        results = set()\n",
    "        while len(results) != 5:\n",
    "            c = np.random.choice(3883, 1, p=p)[0]\n",
    "            results.add(c)\n",
    "        for val in (results):\n",
    "            print(val)\n",
    "            print(movies_orig[val])\n",
    "        \n",
    "        return results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "您看的电影是：[1401 'Ghosts of Mississippi (1996)' 'Drama']\n",
      "以下是给您的推荐：\n",
      "3385\n",
      "[3454 'Whatever It Takes (2000)' 'Comedy|Romance']\n",
      "707\n",
      "[716 'Switchblade Sisters (1975)' 'Crime']\n",
      "2351\n",
      "[2420 'Karate Kid, The (1984)' 'Drama']\n",
      "2189\n",
      "[2258 'Master Ninja I (1984)' 'Action']\n",
      "2191\n",
      "[2260 'Wisdom (1986)' 'Action|Crime']\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{707, 2189, 2191, 2351, 3385}"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "recommend_same_type_movie(1401, 20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 推荐您喜欢的电影\n",
    "思路是使用用户特征向量与电影特征矩阵计算所有电影的评分，取评分最高的top_k个，同样加了些随机选择部分。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "def recommend_your_favorite_movie(user_id_val, top_k = 10):\n",
    "\n",
    "    loaded_graph = tf.Graph()  #\n",
    "    with tf.Session(graph=loaded_graph) as sess:  #\n",
    "        # Load saved model\n",
    "        loader = tf.train.import_meta_graph(load_dir + '.meta')\n",
    "        loader.restore(sess, load_dir)\n",
    "\n",
    "        #推荐您喜欢的电影\n",
    "        probs_embeddings = (users_matrics[user_id_val-1]).reshape([1, 200])\n",
    "\n",
    "        probs_similarity = tf.matmul(probs_embeddings, tf.transpose(movie_matrics))\n",
    "        sim = (probs_similarity.eval())\n",
    "    #     print(sim.shape)\n",
    "    #     results = (-sim[0]).argsort()[0:top_k]\n",
    "    #     print(results)\n",
    "        \n",
    "    #     sim_norm = probs_norm_similarity.eval()\n",
    "    #     print((-sim_norm[0]).argsort()[0:top_k])\n",
    "    \n",
    "        print(\"以下是给您的推荐：\")\n",
    "        p = np.squeeze(sim)\n",
    "        p[np.argsort(p)[:-top_k]] = 0\n",
    "        p = p / np.sum(p)\n",
    "        results = set()\n",
    "        while len(results) != 5:\n",
    "            c = np.random.choice(3883, 1, p=p)[0]\n",
    "            results.add(c)\n",
    "        for val in (results):\n",
    "            print(val)\n",
    "            print(movies_orig[val])\n",
    "\n",
    "        return results\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "以下是给您的推荐：\n",
      "1642\n",
      "[1688 'Anastasia (1997)' \"Animation|Children's|Musical\"]\n",
      "994\n",
      "[1007 'Apple Dumpling Gang, The (1975)' \"Children's|Comedy|Western\"]\n",
      "667\n",
      "[673 'Space Jam (1996)' \"Adventure|Animation|Children's|Comedy|Fantasy\"]\n",
      "1812\n",
      "[1881 'Quest for Camelot (1998)' \"Adventure|Animation|Children's|Fantasy\"]\n",
      "1898\n",
      "[1967 'Labyrinth (1986)' \"Adventure|Children's|Fantasy\"]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{667, 994, 1642, 1812, 1898}"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "recommend_your_favorite_movie(234, 10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 看过这个电影的人还看了（喜欢）哪些电影\n",
    "- 首先选出喜欢某个电影的top_k个人，得到这几个人的用户特征向量。\n",
    "- 然后计算这几个人对所有电影的评分\n",
    "- 选择每个人评分最高的电影作为推荐\n",
    "- 同样加入了随机选择"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "\n",
    "def recommend_other_favorite_movie(movie_id_val, top_k = 20):\n",
    "    loaded_graph = tf.Graph()  #\n",
    "    with tf.Session(graph=loaded_graph) as sess:  #\n",
    "        # Load saved model\n",
    "        loader = tf.train.import_meta_graph(load_dir + '.meta')\n",
    "        loader.restore(sess, load_dir)\n",
    "\n",
    "        probs_movie_embeddings = (movie_matrics[movieid2idx[movie_id_val]]).reshape([1, 200])\n",
    "        probs_user_favorite_similarity = tf.matmul(probs_movie_embeddings, tf.transpose(users_matrics))\n",
    "        favorite_user_id = np.argsort(probs_user_favorite_similarity.eval())[0][-top_k:]\n",
    "    #     print(normalized_users_matrics.eval().shape)\n",
    "    #     print(probs_user_favorite_similarity.eval()[0][favorite_user_id])\n",
    "    #     print(favorite_user_id.shape)\n",
    "    \n",
    "        print(\"您看的电影是：{}\".format(movies_orig[movieid2idx[movie_id_val]]))\n",
    "        \n",
    "        print(\"喜欢看这个电影的人是：{}\".format(users_orig[favorite_user_id-1]))\n",
    "        probs_users_embeddings = (users_matrics[favorite_user_id-1]).reshape([-1, 200])\n",
    "        probs_similarity = tf.matmul(probs_users_embeddings, tf.transpose(movie_matrics))\n",
    "        sim = (probs_similarity.eval())\n",
    "    #     results = (-sim[0]).argsort()[0:top_k]\n",
    "    #     print(results)\n",
    "    \n",
    "    #     print(sim.shape)\n",
    "    #     print(np.argmax(sim, 1))\n",
    "        p = np.argmax(sim, 1)\n",
    "        print(\"喜欢看这个电影的人还喜欢看：\")\n",
    "\n",
    "        results = set()\n",
    "        while len(results) != 5:\n",
    "            c = p[random.randrange(top_k)]\n",
    "            results.add(c)\n",
    "        for val in (results):\n",
    "            print(val)\n",
    "            print(movies_orig[val])\n",
    "        \n",
    "        return results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "您看的电影是：[1401 'Ghosts of Mississippi (1996)' 'Drama']\n",
      "喜欢看这个电影的人是：[[5782 'F' 35 0]\n",
      " [5767 'M' 25 2]\n",
      " [3936 'F' 35 12]\n",
      " [3595 'M' 25 0]\n",
      " [1696 'M' 35 7]\n",
      " [2728 'M' 35 12]\n",
      " [763 'M' 18 10]\n",
      " [4404 'M' 25 1]\n",
      " [3901 'M' 18 14]\n",
      " [371 'M' 18 4]\n",
      " [1855 'M' 18 4]\n",
      " [2338 'M' 45 17]\n",
      " [450 'M' 45 1]\n",
      " [1130 'M' 18 7]\n",
      " [3035 'F' 25 7]\n",
      " [100 'M' 35 17]\n",
      " [567 'M' 35 20]\n",
      " [5861 'F' 50 1]\n",
      " [4800 'M' 18 4]\n",
      " [3281 'M' 25 17]]\n",
      "喜欢看这个电影的人还喜欢看：\n",
      "1779\n",
      "[1848 'Borrowers, The (1997)' \"Adventure|Children's|Comedy|Fantasy\"]\n",
      "1244\n",
      "[1264 'Diva (1981)' 'Action|Drama|Mystery|Romance|Thriller']\n",
      "1812\n",
      "[1881 'Quest for Camelot (1998)' \"Adventure|Animation|Children's|Fantasy\"]\n",
      "1742\n",
      "[1805 'Wild Things (1998)' 'Crime|Drama|Mystery|Thriller']\n",
      "2535\n",
      "[2604 'Let it Come Down: The Life of Paul Bowles (1998)' 'Documentary']\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{1244, 1742, 1779, 1812, 2535}"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "recommend_other_favorite_movie(1401, 20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 结论\n",
    "\n",
    "以上就是实现的常用的推荐功能，将网络模型作为回归问题进行训练，得到训练好的用户特征矩阵和电影特征矩阵进行推荐。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## 扩展阅读\n",
    "如果你对个性化推荐感兴趣，以下资料建议你看看："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- [`Understanding Convolutional Neural Networks for NLP`](http://www.wildml.com/2015/11/understanding-convolutional-neural-networks-for-nlp/)\n",
    "\n",
    "- [`Convolutional Neural Networks for Sentence Classification`](https://github.com/yoonkim/CNN_sentence)\n",
    "\n",
    "- [`利用TensorFlow实现卷积神经网络做文本分类`](http://www.jianshu.com/p/ed3eac3dcb39?from=singlemessage)\n",
    "\n",
    "- [`Convolutional Neural Network for Text Classification in Tensorflow`](https://github.com/dennybritz/cnn-text-classification-tf)\n",
    "\n",
    "- [`SVD Implement Recommendation systems`](https://github.com/songgc/TF-recomm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "今天的分享就到这里，请多指教！"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
